Video coding using camera motion compensation and object motion compensation

ABSTRACT

Systems and techniques are provided for coding (e.g., encoding and/or decoding) video data using camera motion information. For example, a decoding device can obtain a frame of encoded video data associated with an input frame, the frame of encoded video data including camera information associated with generating the video data and a residual. A camera motion compensated frame can be generated based on a reference frame and the camera information. Optical flow information associated with object motion determined based on at least the input frame and the reference frame can be generated. A motion compensated frame can be generated by warping the camera motion compensated frame based on the optical flow information. A reconstructed input frame can be generated based on the motion compensated frame and the residual.

FIELD

The present disclosure generally relates to video coding (e.g., encodingand/or decoding video data). For example, aspects of the presentdisclosure are related to systems and techniques for performing cameramotion compensation and object motion compensation for video coding.

BACKGROUND

Many devices and systems allow video data to be processed and output forconsumption. Digital video data includes large amounts of data to meetincreasing demands in video quality, performance, and features. Forexample, consumers of video data typically desire high quality videos,with high fidelity, resolutions, frame rates, and the like. As a result,the large amount of video data often needed to meet these demands placesa burden on communication networks and devices that process and storethe video data.

Video coding techniques may be used to compress video data. One examplegoal of video coding is to compress video data into a form that uses alower bit rate, while avoiding or minimizing degradations in videoquality. With ever-evolving video services becoming available and theincreasing demands in large amounts of video data, coding techniqueswith better performance and efficiency are needed.

SUMMARY

In some examples, systems and techniques are described for coding (e.g.,encoding and/or decoding) video data using a machine learningarchitecture. For example, the systems and techniques can use one ormore machine learning networks (e.g., neural networks) to perform videocoding for rendered video data and/or other video data, based on depthinformation and/or camera pose information associated with the renderedvideo data. According to at least one illustrative example, an apparatusfor decoding video data is provided that includes at least one memory(e.g., configured to store data, such as virtual content data, one ormore images, etc.) and one or more processors (e.g., implemented incircuitry) coupled to the at least one memory. The one or moreprocessors are configured to and can: obtain a frame of encoded videodata associated with an input frame, the frame of encoded video dataincluding camera information associated with generating the video dataand a residual; generate a camera motion compensated frame based on areference frame and the camera information; generate optical flowinformation associated with object motion determined based on at leastthe input frame and the reference frame; generate a motion compensatedframe by warping the camera motion compensated frame based on theoptical flow information; and generate, based on the motion compensatedframe and the residual, a reconstructed input frame.

In another example, a method for decoding video data is provided, themethod including: obtaining a frame of encoded video data associatedwith an input frame, the frame of encoded video data including camerainformation associated with generating the video data and a residual;generating a camera motion compensated frame based on a reference frameand the camera information; generating optical flow informationassociated with object motion determined based on at least the inputframe and the reference frame; generating a motion compensated frame bywarping the camera motion compensated frame based on the optical flowinformation; and generating, based on the motion compensated frame andthe residual, a reconstructed input frame.

In another example, a non-transitory computer-readable medium isprovided that has stored thereon instructions that, when executed by oneor more processors, cause the one or more processors to: obtain a frameof encoded video data associated with an input frame, the frame ofencoded video data including camera information associated withgenerating the video data and a residual; generate a camera motioncompensated frame based on a reference frame and the camera information;generate optical flow information associated with object motiondetermined based on at least the input frame and the reference frame;generate a motion compensated frame by warping the camera motioncompensated frame based on the optical flow information; and generate,based on the motion compensated frame and the residual, a reconstructedinput frame.

In another example, an apparatus for decoding video data is provided.The apparatus includes: means for obtaining a frame of encoded videodata associated with an input frame, the frame of encoded video dataincluding camera information associated with generating the video dataand a residual; means for generating a camera motion compensated framebased on a reference frame and the camera information; means forgenerating optical flow information associated with object motiondetermined based on at least the input frame and the reference frame;means for generating a motion compensated frame by warping the cameramotion compensated frame based on the optical flow information; andmeans for generating, based on the motion compensated frame and theresidual, a reconstructed input frame.

In another example, an apparatus for encoding video data is providedthat includes at least one memory (e.g., configured to store data, suchas virtual content data, one or more images, etc.) and one or moreprocessors (e.g., implemented in circuitry) coupled to the at least onememory. The one or more processors are configured to and can: obtain aninput frame of video data and camera information associated withgenerating the input frame of video data; generate a camera motioncompensated frame based on a reference frame and the camera information;generate optical flow information associated with object motiondetermined based on at least the input frame and the reference frame;generate a motion compensated frame by warping the camera motioncompensated frame based on the optical flow information; determine,based on a difference between the input frame and a reconstructed inputframe generated using the motion compensated frame, a residual; andgenerate a frame of encoded video data associated with the input frameof video data, the frame of encoded video data including the camerainformation and the residual.

In another example, a method for encoding video data is provided, themethod including: obtaining an input frame of video data and camerainformation associated with generating the input frame of video data;generating a camera motion compensated frame based on a reference frameand the camera information; generating optical flow informationassociated with object motion determined based on at least the inputframe and the reference frame; generating a motion compensated frame bywarping the camera motion compensated frame based on the optical flowinformation; determining, based on a difference between the input frameand a reconstructed input frame generated using the motion compensatedframe, a residual; and generating a frame of encoded video dataassociated with the input frame of video data, the frame of encodedvideo data including the camera information and the residual.

In another example, a non-transitory computer-readable medium isprovided that has stored thereon instructions that, when executed by oneor more processors, cause the one or more processors to: obtain an inputframe of video data and camera information associated with generatingthe input frame of video data; generate a camera motion compensatedframe based on a reference frame and the camera information; generateoptical flow information associated with object motion determined basedon at least the input frame and the reference frame; generate a motioncompensated frame by warping the camera motion compensated frame basedon the optical flow information; determine, based on a differencebetween the input frame and a reconstructed input frame generated usingthe motion compensated frame, a residual; and generate a frame ofencoded video data associated with the input frame of video data, theframe of encoded video data including the camera information and theresidual.

In another example, an apparatus for encoding video data is provided.The apparatus includes: means for obtaining an input frame of video dataand camera information associated with generating the input frame ofvideo data; means for generating a camera motion compensated frame basedon a reference frame and the camera information; means for generatingoptical flow information associated with object motion determined basedon at least the input frame and the reference frame; means forgenerating a motion compensated frame by warping the camera motioncompensated frame based on the optical flow information; means fordetermining, based on a difference between the input frame and areconstructed input frame generated using the motion compensated frame,a residual; and means for generating a frame of encoded video dataassociated with the input frame of video data, the frame of encodedvideo data including the camera information and the residual.

In some aspects, the apparatus can include or be part of a mobiledevice, a wearable device, an extended reality device (e.g., a virtualreality (VR) device, an augmented reality (AR) device, or a mixedreality (MR) device), a personal computer, a laptop computer, a servercomputer, a television, a video game console, or other device. In someaspects, the apparatus comprises a mobile device (e.g., a mobiletelephone or so-called “smart phone”). In some aspects, the apparatusfurther includes at least one camera for capturing one or more images orvideo frames. For example, the apparatus can include a camera (e.g., anRGB camera) or multiple cameras for capturing one or more images and/orone or more videos including video frames. In some aspects, theapparatus includes a display for displaying one or more images, videos,notifications, or other displayable data. In some aspects, the apparatusincludes a transmitter configured to transmit the reconstructed videoframe over a transmission medium to at least one device. In someaspects, the processor includes a neural processing unit (NPU), acentral processing unit (CPU), a graphics processing unit (GPU), orother processing device or component.

This summary is not intended to identify key or essential features ofthe claimed subject matter, nor is it intended to be used in isolationto determine the scope of the claimed subject matter. The subject mattershould be understood by reference to appropriate portions of the entirespecification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and aspects, will becomemore apparent upon referring to the following specification, claims, andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative aspects of the present application are described in detailbelow with reference to the following figures:

FIG. 1 illustrates an example image processing system which canimplement the various techniques described herein, in accordance withsome examples;

FIG. 2A illustrates an example of a fully connected neural network, inaccordance with some examples;

FIG. 2B illustrates an example of a locally connected neural network, inaccordance with some examples;

FIG. 2C illustrates an example of a convolutional neural network, inaccordance with some examples;

FIG. 2D illustrates a detailed example of a deep convolutional network(DCN) designed to recognize features from an image, in accordance withsome examples;

FIG. 3 is a block diagram illustrating another example DCN, inaccordance with some examples;

FIG. 4 is a diagram illustrating an example of a system including adevice operable to perform image and/or video coding (e.g., encoding anddecoding) using a neural network-based system, in accordance with someexamples;

FIG. 5 is a diagram illustrating examples of motion estimationtechniques, in accordance with some examples;

FIG. 6 is a diagram illustrating an example neural network-based videocodec (e.g., coder/decoder) that can be used to perform video coding forrendered content, in accordance with some examples;

FIG. 7 is a diagram illustrating an example of a neural video codingsystem that can be used to perform unidirectional coding and/orbidirectional coding, in accordance with some examples;

FIG. 8 is a diagram illustrating an example of a machine learning-basedvideo coding system including a motion compensation sub-network and anobject motion compensation sub-network, in accordance with someexamples;

FIG. 9 is a diagram illustrating another example of a machinelearning-based video coding system including a motion compensationsub-network and an object motion compensation sub-network, in accordancewith some examples;

FIG. 10 is a diagram illustrating another example of a machinelearning-based video coding system including a motion compensationsub-network and an object motion compensation sub-network, in accordancewith some examples;

FIG. 11 is a flowchart illustrating an example of a process forprocessing video data, in accordance with some examples; and

FIG. 12 illustrates an example computing system that can be used toimplement various aspects described herein.

DETAILED DESCRIPTION

Certain aspects of this disclosure are provided below. Some of theseaspects may be applied independently and some of them may be applied incombination as would be apparent to those of skill in the art. In thefollowing description, for the purposes of explanation, specific detailsare set forth in order to provide a thorough understanding of aspects ofthe application. However, it will be apparent that various aspects maybe practiced without these specific details. The figures and descriptionare not intended to be restrictive.

The ensuing description provides example aspects only, and is notintended to limit the scope, applicability, or configuration of thedisclosure. Rather, the ensuing description of the example aspects willprovide those skilled in the art with an enabling description forimplementing an example aspect. It should be understood that variouschanges may be made in the function and arrangement of elements withoutdeparting from the spirit and scope of the application as set forth inthe appended claims

Digital video data can include large amounts of data, particularly asthe demand for high quality video data continues to grow. For example,consumers of video data typically desire video of increasingly highquality, with high fidelity, resolution, frame rates, and the like.However, the large amount of video data often needed to meet suchdemands can place a significant burden on communication networks as welldevices that process and store the video data.

Various techniques can be used to code video data. In some cases, videocoding can be performed according to a particular video coding standardand/or scheme. Example video coding standards include high-efficiencyvideo coding (HEVC), advanced video coding (AVC), moving picture expertsgroup (MPEG) coding, versatile video coding (VVC), among others. Onegoal of video coding techniques is to compress video data into a formthat uses a lower bit rate, while avoiding or minimizing degradations inthe video quality. As the demand for video services grows and new videoservices become available, coding techniques with better efficiency andperformance are needed.

Video coding can use prediction methods such as intra-prediction orinter-prediction, which take advantage of redundancies present in videoframes or other sequences of images or frames. Intra-prediction isperformed using the data within a single frame of video, and is based onthe spatial characteristics of the frame. Frames coded usingintra-prediction are referred to as I-frames. Inter-prediction of aframe is performed based on temporal characteristics of the framerelative to other frames. For example, inter-prediction of a video framecan be performed by identifying regions of other video frames thatinclude changes relative to the video frame and regions that includeredundancies relative to the video frame (e.g., background regions thatremain largely unchanged). The redundancies can be removed, resulting ina residual for the video frame. The residual can be further encoded(e.g., using entropy coding), and the result can be included in abitstream that is stored, transmitted, or otherwise output.

Examples of inter-prediction include unidirectional prediction(uni-prediction) and bidirectional prediction (bi-prediction).Uni-prediction includes the use of a single reference frame whenperforming inter-prediction of a frame. Frames coded usinguni-prediction are referred to as predicted frames (P-frames).Bi-prediction involves the use of two reference frames when performinginter-prediction of a frame. Frames coded using bi-prediction arereferred to as bi-predicted frames (B-frames).

In some cases, machine learning systems can be used to perform videoencoding (compression) and decoding (decompression). In general, machinelearning (ML) is a subset of artificial intelligence (AI). ML systemsinclude algorithms and statistical models that computer systems can useto perform various tasks by relying on patterns and inference. Oneexample of a ML system is a neural network (also referred to as anartificial neural network), which can include an interconnected group ofartificial neurons (e.g., neuron models). Neural networks may be usedfor various applications and/or devices, such as image analysis and/orcomputer vision applications, Internet Protocol (IP) cameras, Internetof Things (IoT) devices, autonomous vehicles, service robots, amongothers.

Individual nodes in the neural network may emulate biological neurons bytaking input data and performing operations on the data. The results ofthe operations performed on the input data are selectively passed toother neurons. Weight values are associated with each vector and node inthe network, and these values constrain how input data is related tooutput data. For example, the input data of each node may be multipliedby a corresponding weight value, and the products may be summed. The sumof the products may be adjusted by an optional bias, and an activationfunction may be applied to the result, yielding the node's output signalor “output activation” (sometimes referred to as an activation map orfeature map). The weight values may initially be determined by aniterative flow of training data through the network. For instance,weight values may be established during a training phase in which thenetwork learns how to identify particular classes by their typical inputdata characteristics. In one example, the network may be trained tolearn a particular task by adapting values of parameters associated withthe neurons (e.g., activation parameters and/or weights, biases, etc.),adding and/or removing neurons or even layers of neurons, adding orremoving edges between neurons, etc.

Different types of neural networks exist, such as autoencoders,convolutional neural networks (CNNs), recurrent neural networks (RNNs),multilayer perceptron (MLP) neural networks, among others. Convolutionalneural networks may include collections of artificial neurons that eachhave a receptive field (e.g., a spatially localized region of an inputspace) and that collectively tile an input space. Convolutional neuralnetworks have numerous applications. For example, CNNs can be broadlyused in the area of pattern recognition and classification. RNNs work onthe principle of saving the output of a layer and feeding this outputback to the input to help in predicting an outcome of the layer. In MLPneural networks, data may be fed into an input layer, and one or morehidden layers provide levels of abstraction to the data. Predictions maythen be made on an output layer based on the abstracted data. MLPs maybe particularly suitable for classification prediction problems whereinputs are assigned a class or label.

In layered neural network architectures (referred to as deep neuralnetworks when multiple hidden layers are present), the output of a firstlayer of artificial neurons becomes an input to a second layer ofartificial neurons, the output of a second layer of artificial neuronsbecomes an input to a third layer of artificial neurons, and so on.Convolutional neural networks may be trained to recognize a hierarchy offeatures. Computation in convolutional neural network architectures maybe distributed over a population of processing nodes, which may beconfigured in one or more computational chains. These multi-layeredarchitectures may be trained one layer at a time and may be fine-tunedusing back propagation.

In some examples, machine learning based P-frame and B-frame codingsystems can be used to perform uni-prediction and bi-prediction,respectively. In some cases, such systems can include neural networkarchitectures (e.g., one or more deep neural networks, such as one ormore autoencoders). An example of a machine learning based P-framecoding system can perform motion compression and motion compensation ona current frame and a reference frame to determine a prediction ofmotion between the current frame and the reference frame. The motionprediction can be used to modify the pixels of the reference frame(e.g., by moving the pixels of the reference frame according to motionvectors included in the motion prediction), resulting in a predictionfor the current frame. A residual portion of the P-frame coding systemcan generate a predicted residual representing a difference between theprediction and the current frame. The predicted residual can be combinedwith the prediction to generate a reconstructed current frame.

Systems, methods (also referred to as processes), apparatuses, andcomputer-readable media (collectively referred to as “systems andtechniques”) are described herein for performing coding of video databased on using rendering information and/or additional informationassociated with the video data to perform camera motion compensation andobject motion compensation. As used herein, the term coding can refer toencoding (e.g., compression), decoding (e.g., decompression), or bothencoding and decoding. For example, the systems and techniques describedherein can perform video coding for frames of video content (e.g.,frames of rendered and/or gaming content) based on using separate motionprediction systems to perform camera motion compensation and to performobject motion compensation. In some cases, the systems and techniquescan perform camera motion compensation based on rendering informationassociated with one or more frames of the video content.

In some aspects, rendering information associated with one or moreframes of the currently encoded and/or decoded video content can includea current camera pose p_(t) (e.g., the camera pose associated with thecurrent frame t), a reference camera pose p_(t-1) (e.g., the camera poseassociated with the prior frame t−1), and a depth map D_(t) associatedwith the current frame t. In some cases, camera pose information caninclude position and orientation information of a camera with respect toa reference coordinate system of the scene depicted in the video content(e.g., the scene captured by the camera). A depth map can be an imagethat includes information associated with the distance of the surfacesof scene objects from a given viewpoint (e.g., the viewpoint of thecamera that captures the scene). In some examples, a depth map can havethe same pixel dimensions as a corresponding frame of video data forwhich the depth map is generated, wherein some (or all) of the pixelsincluded in the depth map are associated with a depth value. The depthvalue associated with a given pixel can indicate a distance from thecamera viewpoint to the surface of the scene object depicted by thegiven pixel.

In examples in which the frames of encoded and/or decoded video contentare frames of rendered video data, the camera pose information and depthinformation may be generated in association with rendering the videoframes (e.g., generated at the time the video frame is originallygenerated, prior to encoding or decoding). For example, camera pose anddepth information can be obtained from one or more rendering enginesassociated with generating the rendered frames, based on the renderingengines having used the camera pose and depth information to originallygenerate the rendered frames. In some aspects, camera pose and depthinformation associated with frames of rendered gaming content can beobtained from one or more buffers associated with a rendering or gameengine used to generate the gaming content. For example, camera pose anddepth information associated with frames of rendered gaming contentgenerated by a game engine and/or a cloud gaming server of can beobtained from a deferred rendering buffer and/or geometry buffer(G-buffer) associated with the game engine. In some examples, the camerapose and depth information may be stored in a deferred rendering bufferor G-buffer among various other types of rendering information (e.g.,motion information, optical flow maps, normal maps, albedo maps, etc.).

In some examples, the systems and techniques described herein can beused to perform video coding based on separate camera motioncompensation and object motion compensation for non-rendered videocontent (e.g., natural video content). For example, the camera poseinformation p_(t) and p_(t-1) can be obtained from a camera used tocaptured the non-rendered or natural video content, can be determinedusing one or more post-processing algorithms or machine-learningnetworks, etc. In some cases, depth information can be obtained from acamera that includes one or more depth sensors and captures depthinformation and video content simultaneously (e.g., such that the videocontent is captured in combination with a corresponding depth map forone or more frames of the captured video content). In some aspects,depth information can be obtained from a stereo camera that includes twoor more cameras for determining depth maps.

In some examples, a depth map prediction system and a three-dimensional(3D) warping engine can be used to generate an initial frame predictionX _(t) that is compensated using camera motion determined based on theinput camera pose p_(t), reference camera pose p_(t-1), and depth mapD_(t). In some cases, the initial frame prediction X _(t) may also bereferred to as a camera motion compensated frame. A 3D warping enginecan generate the initial frame prediction X _(t) based on receiving asinput the camera pose p_(t) for the current frame t, the referencecamera pose p_(t-1) for the previous frame t−1, the previously decoded(e.g. reconstructed) frame {circumflex over (X)}_(t-1), and a predicted(e.g., reconstructed) depth map {circumflex over (d)}_(t) for thecurrent frame t. The camera poses p_(t) and p_(t-1), along with thedepth map D_(t) can be generated by a rendering engine (e.g., a cloudgaming engine) associated with a neural video encoder. In some examples,the neural video encoder can be included in a same server or othercomputing device as the rendering engine used to generate the frames ofrendered video data being encoded (e.g., which may also be the samerendering engine used to generate the camera poses p_(t) and p_(t-1),along with the depth map D_(t)).

Further aspects of the systems and techniques will be described withrespect to the figures. FIG. 1 illustrates an example implementation ofan image processing system 100 that, in some cases, can be used toimplement the systems and techniques described herein. The imageprocessing system 100 can include a central processing unit (CPU) 102 ora multi-core CPU, configured to perform one or more of the functionsdescribed herein. Parameters or variables (e.g., neural signals andsynaptic weights), system parameters associated with a computationaldevice (e.g., neural network with weights), delays, frequency bininformation, task information, image data, among other information maybe stored in a memory block associated with a neural processing unit(NPU) 108, in a memory block associated with a CPU 102, in a memoryblock associated with a graphics processing unit (GPU) 104, in a memoryblock associated with a digital signal processor (DSP) 106, in a memoryblock 118, and/or may be distributed across multiple blocks.Instructions executed at the CPU 102 may be loaded from a program memoryassociated with the CPU 102 and/or from a memory block 118.

The image processing system 100 can also include additional processingblocks for performing specific functions, such as a GPU 104; a DSP 106;a connectivity block 110, which may include fifth generation (5G)connectivity, fourth generation long term evolution (4G LTE)connectivity, Wi-Fi connectivity, USB connectivity, Bluetoothconnectivity, and the like; and/or a multimedia processor 112 that may,for example, detect image features. In some examples, the NPU 108 can beimplemented in the CPU 102, DSP 106, and/or GPU 104. In some cases, theimage processing system 100 may also include one or more sensor 114, oneor more image signal processors (ISPs) 116, and/or storage 120.

In some examples, the image processing system 100 can implement an ARMinstruction set architecture for one or more processors. In an aspect ofthe present disclosure, the instructions loaded into the CPU 102 mayinclude code to search for a stored multiplication result in a lookuptable (LUT) corresponding to a multiplication product of an input valueand a filter weight. The instructions loaded into the CPU 102 may alsoinclude code to disable a multiplier during a multiplication operationof the multiplication product when a lookup table hit of themultiplication product is detected. In addition, the instructions loadedinto the CPU 102 may include code to store a computed multiplicationproduct of the input value and the filter weight when a lookup tablemiss of the multiplication product is detected.

The image processing system 100 can be part of a computing device ormultiple computing devices. In some examples, the image processingsystem 100 can be part of an electronic device (or devices) such as acamera system (e.g., a digital camera, an IP camera, a video camera, asecurity camera, etc.), a telephone system (e.g., a smartphone, acellular telephone, a conferencing system, etc.), a desktop computer, anXR device (e.g., a head-mounted display, etc.), a smart wearable device(e.g., a smart watch, smart glasses, etc.), a laptop or notebookcomputer, a tablet computer, a set-top box, a television, a displaydevice, a system-on-chip (SoC), a digital media player, a gamingconsole, a video streaming device, a server, a drone, a computer in acar, an Internet-of-Things (IoT) device, or any other suitableelectronic device(s).

In some implementations, the CPU 102, the GPU 104, the DSP 106, the NPU108, the connectivity block 110, the multimedia processor 112, the oneor more sensors 114, the ISPs 116, the memory block 118 and/or thestorage 120 can be part of the same computing device. For example, insome cases, the CPU 102, the GPU 104, the DSP 106, the NPU 108, theconnectivity block 110, the multimedia processor 112, the one or moresensors 114, the ISPs 116, the memory block 118 and/or the storage 120can be integrated into a smartphone, laptop, tablet computer, smartwearable device, video gaming system, server, and/or any other computingdevice. In other implementations, the CPU 102, the GPU 104, the DSP 106,the NPU 108, the connectivity block 110, the multimedia processor 112,the one or more sensors 114, the ISPs 116, the memory block 118 and/orthe storage 120 can be part of two or more separate computing devices.

The image processing system 100 and/or components thereof may beconfigured to perform video compression and/or decompression (alsoreferred to as video encoding and/or decoding, collectively referred toas video coding) using techniques according to aspects of the presentdisclosure discussed herein. By using deep learning architectures andthe techniques described herein to perform video compression and/ordecompression, aspects of the present disclosure can increase theefficiency of video compression and/or decompression on a device and/orreduce associated resource requirements and/or usage. For example, adevice using the video coding techniques described herein can compressvideo data more efficiently, can reduce the amount of data transmittedin compressed video data to a destination device, and the destinationdevice can receive and decompress the compressed video data efficiently.In some examples, the deep learning architectures and techniquesdescribed herein can reduce the amount of data exchanged between codingdevices or components, such as encoders and decoders, to code videocontent. The reduced amount of data transmitted for video coding canreduce latencies, increase performance, and reduce the cost or burden oncomputing resources such as, for example, bandwidth, memory, storage,power, compute, hardware, etc.

As noted above, a neural network is an example of a machine learningsystem, and can include an input layer, one or more hidden layers, andan output layer. Data is provided from input nodes of the input layer,processing is performed by hidden nodes of the one or more hiddenlayers, and an output is produced through output nodes of the outputlayer. Deep learning networks typically include multiple hidden layers.Each layer of the neural network can include feature maps or activationmaps that can include artificial neurons (or nodes). A feature map caninclude a filter, a kernel, or the like. The nodes can include one ormore weights used to indicate an importance of the nodes of one or moreof the layers. In some cases, a deep learning network can have a seriesof many hidden layers, with early layers being used to determine simpleand low level characteristics of an input, and later layers building upa hierarchy of more complex and abstract characteristics.

A deep learning architecture may learn a hierarchy of features. Ifpresented with visual data, for example, the first layer may learn torecognize features, such as edges, in the input stream. In anotherexample, if presented with auditory data, the first layer may learn torecognize spectral power in specific frequencies. The second layer,taking the output of the first layer as input, may learn to recognizefeatures, such as shapes for visual data or combinations of sounds forauditory data. For instance, higher layers may learn to representcomplex shapes in visual data or words in auditory data. Still higherlayers may learn to recognize common visual objects and/or spokenphrases.

Neural networks may be designed with a variety of connectivity patterns.In feed-forward networks, information is passed from lower to higherlayers, with each neuron in a given layer communicating to neurons inhigher layers. A hierarchical representation may be built up insuccessive layers of a feed-forward network, as described above. Neuralnetworks may also have recurrent or feedback (also called top-down)connections. In a recurrent connection, the output from a neuron in agiven layer may be communicated to another neuron in the same layer. Arecurrent architecture may be helpful in recognizing patterns that spanmore than one of the input data chunks that are delivered to the neuralnetwork in a sequence. A connection from a neuron in a given layer to aneuron in a lower layer is called a feedback (or top-down) connection. Anetwork with many feedback connections may be helpful when therecognition of a high-level concept may aid in discriminating theparticular low-level features of an input.

The connections between layers of a neural network may be fullyconnected or locally connected. FIG. 2A illustrates an example of afully connected neural network 202. In a fully connected neural network202, a neuron in a first hidden layer may communicate its output toevery neuron in a second hidden layer, so that each neuron in the secondlayer will receive input from every neuron in the first layer. FIG. 2Billustrates an example of a locally connected neural network 204. In alocally connected neural network 204, a neuron in a first hidden layermay be connected to a limited number of neurons in a second hiddenlayer. More generally, a locally connected layer of the locallyconnected neural network 204 may be configured so that each neuron in alayer will have the same or a similar connectivity pattern, but withconnections strengths that may have different values (e.g., 210, 212,214, and 216). The locally connected connectivity pattern may give riseto spatially distinct receptive fields in a higher layer, because thehigher layer neurons in a given region may receive inputs that are tunedthrough training to the properties of a restricted portion of the totalinput to the network.

One example of a locally connected neural network is a convolutionalneural network. FIG. 2C illustrates an example of a convolutional neuralnetwork 206. The convolutional neural network 206 may be configured suchthat the connection strengths associated with the inputs for each neuronin the second layer are shared (e.g., 208). Convolutional neuralnetworks may be well suited to problems in which the spatial location ofinputs is meaningful. Convolutional neural network 206 may be used toperform one or more aspects of video compression and/or decompression,according to aspects of the present disclosure.

One type of convolutional neural network is a deep convolutional network(DCN). FIG. 2D illustrates an example of a DCN 200 designed to recognizefeatures from an image 226 input from an image capturing device 230,such as a camera or image sensor. In some examples, the DCN 200 of thecurrent example may be trained to identify visual features in the image226, such as one or more objects or signs in the image 226, for example.

In some examples, the DCN 200 may be trained with supervised learning.During training, the DCN 200 may be presented with an image, such as theimage 226, and a forward pass may then be computed to produce an output222. The DCN 200 may include a feature extraction section and aclassification section. Upon receiving the image 226, a convolutionallayer 232 may apply convolutional kernels (not shown) to the image 226to generate a first set of feature maps 218. As an example, theconvolutional kernel for the convolutional layer 232 may be a 5×5 kernelthat generates 28×28 feature maps. In the present example, because fourdifferent feature maps are generated in the first set of feature maps218, four different convolutional kernels were applied to the image 226at the convolutional layer 232. The convolutional kernels may also bereferred to as filters or convolutional filters.

The first set of feature maps 218 may be subsampled by a max poolinglayer (not shown) to generate a second set of feature maps 220. The maxpooling layer reduces the size of the first set of feature maps 218.That is, a size of the second set of feature maps 220, such as 14×14, isless than the size of the first set of feature maps 218, such as 28×28.The reduced size provides similar information to a subsequent layerwhile reducing memory consumption. The second set of feature maps 220may be further convolved via one or more subsequent convolutional layers(not shown) to generate one or more subsequent sets of feature maps (notshown).

In the example of FIG. 2D, the second set of feature maps 220 isconvolved to generate a first feature vector 224. Furthermore, the firstfeature vector 224 is further convolved to generate a second featurevector 228. Each feature of the second feature vector 228 may include anumber that corresponds to a possible feature of the image 226, such as“sign”, “60”, and “100”. A softmax function (not shown) may convert thenumbers in the second feature vector 228 to a probability. As such, anoutput 222 of the DCN 200 is a probability of the image 226 includingone or more features.

In the present example, the probabilities in the output 222 for “sign”and “60” are higher than the probabilities of the others of the output222, such as “30”, “40”, “50”, “70”, “80”, “90”, and “100”. Beforetraining, the output 222 produced by the DCN 200 is likely to beincorrect. Thus, an error may be calculated between the output 222 and atarget output. The target output is the ground truth of the image 226(e.g., “sign” and “60”). The weights of the DCN 200 may then be adjustedso the output 222 of the DCN 200 is more closely aligned with the targetoutput.

To adjust the weights, a learning algorithm may compute a gradientvector for the weights. The gradient may indicate an amount that anerror would increase or decrease if the weight were adjusted. At the toplayer, the gradient may correspond directly to the value of a weightconnecting an activated neuron in the penultimate layer and a neuron inthe output layer. In lower layers, the gradient may depend on the valueof the weights and on the computed error gradients of the higher layers.The weights may then be adjusted to reduce the error. This manner ofadjusting the weights may be referred to as “back propagation” as itinvolves a “backward pass” through the neural network.

In practice, the error gradient of weights may be calculated over asmall number of examples, so that the calculated gradient approximatesthe true error gradient. This approximation method may be referred to asstochastic gradient descent. Stochastic gradient descent may be repeateduntil the achievable error rate of the entire system has stoppeddecreasing or until the error rate has reached a target level. Afterlearning, the DCN may be presented with new images and a forward passthrough the network may yield an output 222 that may be considered aninference or a prediction of the DCN.

Deep belief networks (DBNs) are probabilistic models comprising multiplelayers of hidden nodes. DBNs may be used to extract a hierarchicalrepresentation of training data sets. A DBN may be obtained by stackingup layers of Restricted Boltzmann Machines (RBMs). An RBM is a type ofartificial neural network that can learn a probability distribution overa set of inputs. Because RBMs can learn a probability distribution inthe absence of information about the class to which each input should becategorized, RBMs are often used in unsupervised learning. Using ahybrid unsupervised and supervised paradigm, the bottom RBMs of a DBNmay be trained in an unsupervised manner and may serve as featureextractors, and the top RBM may be trained in a supervised manner (on ajoint distribution of inputs from the previous layer and target classes)and may serve as a classifier.

Deep convolutional networks (DCNs) are networks of convolutionalnetworks, configured with additional pooling and normalization layers.DCNs can achieve high performance on many tasks. DCNs can be trainedusing supervised learning in which both the input and output targets areknown for many exemplars and are used to modify the weights of thenetwork by use of gradient descent methods.

DCNs may be feed-forward networks. In addition, as described above, theconnections from a neuron in a first layer of a DCN to a group ofneurons in the next higher layer are shared across the neurons in thefirst layer. The feed-forward and shared connections of DCNs may beexploited for fast processing. The computational burden of a DCN may bemuch less than, for example, that of a similarly sized neural networkthat comprises recurrent or feedback connections.

The processing of each layer of a convolutional network may beconsidered a spatially invariant template or basis projection. If theinput is first decomposed into multiple channels, such as the red,green, and blue channels of a color image, then the convolutionalnetwork trained on that input may be considered three-dimensional, withtwo spatial dimensions along the axes of the image and a third dimensioncapturing color information. The outputs of the convolutionalconnections may be considered to form a feature map in the subsequentlayer, with each element of the feature map (e.g., 220) receiving inputfrom a range of neurons in the previous layer (e.g., feature maps 218)and from each of the multiple channels. The values in the feature mapmay be further processed with a non-linearity, such as a rectification,max(0,x). Values from adjacent neurons may be further pooled, whichcorresponds to down sampling, and may provide additional localinvariance and dimensionality reduction.

FIG. 3 is a block diagram illustrating an example of a deepconvolutional network 350. The deep convolutional network 350 mayinclude multiple different types of layers based on connectivity andweight sharing. As shown in FIG. 3 , the deep convolutional network 350includes the convolution blocks 354A, 354B. Each of the convolutionblocks 354A, 354B may be configured with a convolution layer (CONV) 356,a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL)360.

The convolution layers 356 may include one or more convolutionalfilters, which may be applied to the input data 352 to generate afeature map. Although only two convolution blocks 354A, 354B are shown,the present disclosure is not so limiting, and instead, any number ofconvolution blocks (e.g., blocks 354A, 354B) may be included in the deepconvolutional network 350 according to design preferences. Thenormalization layer 358 may normalize the output of the convolutionfilters. For example, the normalization layer 358 may provide whiteningor lateral inhibition. The max pooling layer 360 may provide downsampling aggregation over space for local invariance and dimensionalityreduction.

The parallel filter banks, for example, of a deep convolutional networkmay be loaded on a CPU 102 or GPU 104 of an image processing system 100to achieve high performance and low power consumption. In some examples,the parallel filter banks may be loaded on the DSP 106 or an ISP 116 ofan image processing system 100. The deep convolutional network 350 mayaccess other processing blocks that may be present on the imageprocessing system 100.

The deep convolutional network 350 may include one or more fullyconnected layers, such as layer 362A (labeled “FC1”) and layer 362B(labeled “FC2”). The deep convolutional network 350 may include alogistic regression (LR) layer 364. Between each layer 356, 358, 360,362, 364 of the deep convolutional network 350 are weights (not shown)that are to be updated. The output of each of the layers (e.g., 356,358, 360, 362, 364) may serve as an input of a succeeding one of thelayers (e.g., 356, 358, 360, 362, 364) in the deep convolutional network350 to learn hierarchical feature representations from input data 352(e.g., images, audio, video, sensor data and/or other input data)supplied at the first of the convolution blocks 354A. The output of thedeep convolutional network 350 is a classification score 366 for theinput data 352. The classification score 366 may be a set ofprobabilities, where each probability is the probability of the inputdata including a feature from a set of features.

Another type of neural network is an autoencoder. An autoencoder can betrained (e.g., using training data and one or more loss functions) toreceive input and to generate a version of that input at its output(e.g., to essentially copy its input to its output). An autoencoder canbe trained to learn efficient data codings in an unsupervised manner.For example, given an image of an object, an autoencoder can firstencode the image into a lower dimensional latent representation, and canthen decode the latent representation back to an image of the object. Anautoencoder can learn (through training) to compress the input datawhile minimizing the reconstruction error.

As noted previously, digital video data can include large amounts ofdata, which can place a significant burden on communication networks anddevices that process and store the video data. For instance, recordinguncompressed video content generally results in large file sizes thatgreatly increase as the resolution of the recorded video contentincreases. In one illustrative example, uncompressed 16-bit per channelvideo recorded in 1080p/24 (e.g., a resolution of 1920 pixels in widthand 1080 pixels in height, with 24 frames per second captured) mayoccupy 12.4 megabytes per frame, or 297.6 megabytes per second.Uncompressed 16-bit per channel video recorded in 4K resolution at 24frames per second may occupy 49.8 megabytes per frame, or 1195.2megabytes per second.

Network bandwidth is another constraint for which large video files canbecome problematic. For example, video content is oftentimes deliveredover wireless networks (e.g., via LTE, LTE-Advanced, New Radio (NR),WiFi™, Bluetooth™, or other wireless networks), and can make up a largeportion of consumer internet traffic. Thus, it is desirable to reducethe amount of bandwidth used to deliver video content in these networks.

Because uncompressed video content can result in large files that mayinvolve sizable memory for physical storage and considerable bandwidthfor transmission, video coding techniques can be utilized to compressand decompress such video content, as further described herein.

To reduce the size of video content—and thus the amount of storageinvolved to store video content and the amount of bandwidth involved indelivering video content-various video coding techniques can beperformed according to a particular video coding standard and/or scheme,such as HEVC, AVC, MPEG, VVC, among others. Video coding can useprediction methods such as inter-prediction or intra-prediction, whichtake advantage of redundancies present in video images or sequences. Onegoal of video coding techniques is to compress video data into a formthat uses a lower bit rate, while avoiding or minimizing degradations inthe video quality. As the demand for video services grows and new videoservices become available, coding techniques with better codingefficiency, performance, and rate control are needed.

An encoding device can encode video data according to a video codingstandard to generate an encoded video bitstream. In some examples, anencoded video bitstream (or “video bitstream” or “bitstream”) caninclude a series of one or more coded video sequences. The encodingdevice can generate coded representations of pictures by partitioningeach picture into multiple slices. A slice is independent of otherslices so that information in the slice is coded without dependency ondata from other slices within the same picture. A slice includes one ormore slice segments including an independent slice segment and, ifpresent, one or more dependent slice segments that depend on previousslice segments. In HEVC, the slices are partitioned into coding treeblocks (CTBs) of luma samples and chroma samples. A CTB of luma samplesand one or more CTBs of chroma samples, along with syntax for thesamples, are referred to as a coding tree unit (CTU). A CTU may also bereferred to as a “tree block” or a “largest coding unit” (LCU). A CTU isthe basic processing unit for HEVC encoding. A CTU can be split intomultiple coding units (CUs) of varying sizes. A CU contains luma andchroma sample arrays that are referred to as coding blocks (CBs).

The luma and chroma CBs can be further split into prediction blocks(PBs). A PB is a block of samples of the luma component or a chromacomponent that uses the same motion parameters for inter-prediction orintra-block copy (IBC) prediction (when available or enabled for use).The luma PB and one or more chroma PBs, together with associated syntax,form a prediction unit (PU). For inter-prediction, a set of motionparameters (e.g., one or more motion vectors, reference indices, or thelike) is signaled in the bitstream for each PU and is used forinter-prediction of the luma PB and the one or more chroma PBs. Themotion parameters can also be referred to as motion information. A CBcan also be partitioned into one or more transform blocks (TBs). A TBrepresents a square block of samples of a color component on which aresidual transform (e.g., the same two-dimensional transform in somecases) is applied for coding a prediction residual signal. A transformunit (TU) represents the TBs of luma and chroma samples, andcorresponding syntax elements. Transform coding is described in moredetail below.

According to the HEVC standard, transformations may be performed usingTUs. The TUs may be sized based on the size of PUs within a given CU.The TUs may be the same size or smaller than the PUs. In some examples,residual samples corresponding to a CU may be subdivided into smallerunits using a quadtree structure known as residual quad tree (RQT). Leafnodes of the RQT may correspond to TUs. Pixel difference valuesassociated with the TUs may be transformed to produce transformcoefficients. The transform coefficients may then be quantized by theencoding device.

Once the pictures of the video data are partitioned into CUs, theencoding device predicts each PU using a prediction mode. The predictionunit or prediction block is then subtracted from the original video datato get residuals (described below). For each CU, a prediction mode maybe signaled inside the bitstream using syntax data. A prediction modemay include intra-prediction (or intra-picture prediction) orinter-prediction (or inter-picture prediction). Intra-predictionutilizes the correlation between spatially neighboring samples within apicture. For example, using intra-prediction, each PU is predicted fromneighboring image data in the same picture using, for example, DCprediction to find an average value for the PU, planar prediction to fita planar surface to the PU, direction prediction to extrapolate fromneighboring data, or any other suitable types of prediction.Inter-prediction uses the temporal correlation between pictures in orderto derive a motion-compensated prediction for a block of image samples.For example, using inter-prediction, each PU is predicted using motioncompensation prediction from image data in one or more referencepictures (before or after the current picture in output order). Thedecision whether to code a picture area using inter-picture orintra-picture prediction may be made, for example, at the CU level.

In some examples, the one or more slices of a picture are assigned aslice type. Slice types can include an I slice, a P slice, and a Bslice. An I slice (intra-frames, independently decodable) is a slice ofa picture that is only coded by intra-prediction, and therefore isindependently decodable since the I slice requires only the data withinthe frame to predict any prediction unit or prediction block of theslice. A P slice (unidirectional predicted frames) is a slice of apicture that may be coded with intra-prediction and with unidirectionalinter-prediction. Each prediction unit or prediction block within a Pslice is either coded with intra-prediction or inter-prediction. Wheninter-prediction applies, the prediction unit or prediction block ispredicted by one reference picture, and therefore reference samples arefrom one reference region of one frame. A B slice (bidirectionalpredictive frames) is a slice of a picture that may be coded withintra-prediction and inter-prediction (e.g., either bi-prediction oruni-prediction). A prediction unit or prediction block of a B slice maybe bidirectionally-predicted from two reference pictures. Each picturecan contribute a reference region and sample sets of the two referenceregions can be weighted (e.g., with equal weights or with differentweights) to produce the prediction signal of thebidirectionally-predicted block. As explained above, slices of onepicture are independently coded. In some cases, a picture can be codedas just one slice.

After performing prediction using intra- and/or inter-prediction, theencoding device can perform transformation and quantization. Forexample, following prediction, the encoding device may calculateresidual values corresponding to the PU. Residual values may includepixel difference values between the current block of pixels being coded(the PU) and the prediction block used to predict the current block(e.g., the predicted version of the current block). For example, aftergenerating a prediction block (e.g., using inter-prediction orintra-prediction), the encoding device can generate a residual block bysubtracting the prediction block produced by a prediction unit from thecurrent block. The residual block includes a set of pixel differencevalues that quantify differences between pixel values of the currentblock and pixel values of the prediction block. In some examples, theresidual block may be represented in a two-dimensional block format(e.g., a two-dimensional matrix or array of pixel values). In suchexamples, the residual block is a two-dimensional representation of thepixel values.

Any residual data that may be remaining after prediction is performed istransformed using a block transform, which may be based on discretecosine transform, discrete sine transform, an integer transform, awavelet transform, other suitable transform function, or any combinationthereof. In some cases, one or more block transforms (e.g., sizes 32×32,16×16, 8×8, 4×4, or other suitable size) may be applied to residual datain each CU. In some aspects, a TU may be used for the transform andquantization processes implemented by the encoding device. A given CUhaving one or more PUs may also include one or more TUs. As described infurther detail below, the residual values may be transformed intotransform coefficients using the block transforms, and then may bequantized and scanned using TUs to produce serialized transformcoefficients for entropy coding.

The encoding device may perform quantization of the transformcoefficients. Quantization provides further compression by quantizingthe transform coefficients to reduce the amount of data used torepresent the coefficients. For example, quantization may reduce the bitdepth associated with some or all of the coefficients. In one example, acoefficient with an n-bit value may be rounded down to an m-bit valueduring quantization, with n being greater than m.

Once quantization is performed, the coded video bitstream includesquantized transform coefficients, prediction information (e.g.,prediction modes, motion vectors, block vectors, or the like),partitioning information, and any other suitable data, such as othersyntax data. The different elements of the coded video bitstream maythen be entropy encoded by the encoding device. In some examples, theencoding device may utilize a predefined scan order to scan thequantized transform coefficients to produce a serialized vector that canbe entropy encoded. In some examples, encoding device may perform anadaptive scan. After scanning the quantized transform coefficients toform a vector (e.g., a one-dimensional vector), the encoding device mayentropy encode the vector. For example, the encoding device may usecontext adaptive variable length coding, context adaptive binaryarithmetic coding, syntax-based context-adaptive binary arithmeticcoding, probability interval partitioning entropy coding, or anothersuitable entropy encoding technique.

The encoding device can store the encoded video bitstream and/or cansend the encoded video bitstream data over a communications link to areceiving device, which can include a decoding device. The decodingdevice may decode the encoded video bitstream data by entropy decoding(e.g., using an entropy decoder) and extracting the elements of one ormore coded video sequences making up the encoded video data. Thedecoding device may then rescale and perform an inverse transform on theencoded video bitstream data. Residual data is then passed to aprediction stage of the decoding device. The decoding device thenpredicts a block of pixels (e.g., a PU) using intra-prediction,inter-prediction, IBC, and/or other type of prediction. In someexamples, the prediction is added to the output of the inverse transform(the residual data). The decoding device may output the decoded video toa video destination device, which may include a display or other outputdevice for displaying the decoded video data to a consumer of thecontent.

Video coding systems and techniques defined by the various video codingstandards (e.g., the HEVC video coding techniques described above) maybe able to retain much of the information in raw video content and maybe defined a priori based on signal processing and information theoryconcepts. However, in some cases, a machine learning (ML)-based imageand/or video system can provide benefits over non-ML based image andvideo coding systems, such as an end-to-end neural network-based imageand video coding (E2E-NNVC) system. As described above, many E2E-NNVCsystems are designed as combination of an autoencoder sub-network (theencoder sub-network) and a second sub-network responsible for learning aprobabilistic model over quantized latents used for entropy coding. Suchan architecture can be viewed as a combination of a transform plusquantization module (encoder sub-network) and the entropy modellingsub-network module.

FIG. 4 depicts a system 400 that includes a device 402 configured toperform image and/or video encoding and decoding using an E2E-NNVCsystem 410. The device 402 is coupled to a camera 407 and a storagemedium 414 (e.g., a data storage device). In some implementations, thecamera 407 is configured to provide the image data 408 (e.g., a videodata stream) to the processor 404 for encoding by the E2E-NNVC system410. In some implementations, the device 402 can be coupled to and/orcan include multiple cameras (e.g., a dual-camera system, three cameras,or other number of cameras). In some cases, the device 402 can becoupled to a microphone and/or other input device (e.g., a keyboard, amouse, a touch input device such as a touchscreen and/or touchpad,and/or other input device). In some examples, the camera 407, thestorage medium 414, microphone, and/or other input device can be part ofthe device 402.

The device 402 is also coupled to a second device 490 via a transmissionmedium 418, such as one or more wireless networks, one or more wirednetworks, or a combination thereof. For example, the transmission medium418 can include a channel provided by a wireless network, a wirednetwork, or a combination of a wired and wireless network. Thetransmission medium 418 may form part of a packet-based network, such asa local area network, a wide-area network, or a global network such asthe Internet. The transmission medium 418 may include routers, switches,base stations, or any other equipment that may be useful to facilitatecommunication from the source device to the receiving device. A wirelessnetwork may include any wireless interface or combination of wirelessinterfaces and may include any suitable wireless network (e.g., theInternet or other wide area network, a packet-based network, WiFi™,radio frequency (RF), UWB, WiFi-Direct, cellular, Long-Term Evolution(LTE), WiMax™, or the like). A wired network may include any wiredinterface (e.g., fiber, ethernet, powerline ethernet, ethernet overcoaxial cable, digital signal line (DSL), or the like). The wired and/orwireless networks may be implemented using various equipment, such asbase stations, routers, access points, bridges, gateways, switches, orthe like. The encoded video bitstream data may be modulated according toa communication standard, such as a wireless communication protocol, andtransmitted to the receiving device.

The device 402 includes one or more processors (referred to herein as“processor”) 404 coupled to a memory 406, a first interface (“I/F 1”)412, and a second interface (“I/F 2”) 416. The processor 404 isconfigured to receive image data 408 from the camera 407, from thememory 406, and/or from the storage medium 414. The processor 404 iscoupled to the storage medium 414 via the first interface 412 (e.g., viaa memory bus) and is coupled to the transmission medium 418 via thesecond interface 416 (e.g., a network interface device, a wirelesstransceiver and antenna, one or more other network interface devices, ora combination thereof).

The processor 404 includes the E2E-NNVC system 410. The E2E-NNVC system410 includes an encoder portion 462 and a decoder portion 466. In someimplementations, the E2E-NNVC system 410 can include one or moreauto-encoders. The encoder portion 462 is configured to receive inputdata 470 and to process the input data 470 to generate output data 474at least partially based on the input data 470.

In some implementations, the encoder portion 462 of the E2E-NNVC system410 is configured to perform lossy compression of the input data 470 togenerate the output data 474, so that the output data 474 has fewer bitsthan the input data 470. The encoder portion 462 can be trained tocompress input data 470 (e.g., images or video frames) without usingmotion compensation based on any previous representations (e.g., one ormore previously reconstructed frames). For example, the encoder portion462 can compress a video frame using video data only from that videoframe, and without using any data of previously reconstructed frames.Video frames processed by the encoder portion 462 can be referred toherein as intra-predicted frame (I-frames). In some examples, I-framescan be generated using traditional video coding techniques (e.g.,according to HEVC, VVC, MPEG-4, or other video coding Standard). In suchexamples, the processor 404 may include or be coupled with a videocoding device (e.g., an encoding device) configured to performblock-based intra-prediction, such as that described above with respectto the HEVC Standard. In such examples, the E2E-NNVC system 410 may beexcluded from the processor 404.

In some implementations, the encoder portion 462 of the E2E-NNVC system410 can be trained to compress input data 470 (e.g., video frames) usingmotion compensation based on previous representations (e.g., one or morepreviously reconstructed frames). For example, the encoder portion 462can compress a video frame using video data from that video frame andusing data of previously reconstructed frames. Video frames processed bythe encoder portion 462 can be referred to herein as intra-predictedframe (P-frames). The motion compensation can be used to determine thedata of a current frame by describing how the pixels from a previouslyreconstructed frame move into new positions in the current frame alongwith residual information.

As shown, the encoder portion 462 of the E2E-NNVC system 410 can includea neural network 463 and a quantizer 464. The neural network 463 caninclude one or more transformers, one or more convolutional neuralnetworks (CNNs), one or more fully connected neural networks, one ormore gated recurrent units (GRUs), one or more Long Short-Term Memory(LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or moreConvLSTMs, one or more GANs, any combination thereof, and/or other typesof neural network architectures that generate(s) intermediate data 472.The intermediate data 472 is input to the quantizer 464.

The quantizer 464 is configured to perform quantization and in somecases entropy coding of the intermediate data 472 to produce the outputdata 474. The output data 474 can include the quantized (and in somecases entropy coded) data. The quantization operations performed by thequantizer 464 can result in the generation of quantized codes (or datarepresenting quantized codes generated by the E2E-NNVC system 410) fromthe intermediate data 472. The quantization codes (or data representingthe quantized codes) can also be referred to as latent codes or as alatent (denoted as z). The entropy model that is applied to a latent canbe referred to herein as a “prior”. In some examples, the quantizationand/or entropy coding operations can be performed using existingquantization and entropy coding operations that are performed whenencoding and/or decoding video data according to existing video codingstandards. In some examples, the quantization and/or entropy codingoperations can be done by the E2E-NNVC system 410. In one illustrativeexample, the E2E-NNVC system 410 can be trained using supervisedtraining, with residual data being used as input and quantized codes andentropy codes being used as known output (labels) during the training.

The decoder portion 466 of the E2E-NNVC system 410 is configured toreceive the output data 474 (e.g., directly from quantizer 464 and/orfrom the storage medium 414). The decoder portion 466 can process theoutput data 474 to generate a representation 476 of the input data 470at least partially based on the output data 474. In some examples, thedecoder portion 466 of the E2E-NNVC system 410 includes a neural network468 that may include one or more transformers, one or more CNNs, one ormore fully connected neural networks, one or more GRUs, one or more LS™networks, one or more ConvRNNs, one or more ConvGRUs, one or moreConvLSTMs, one or more GANs, any combination thereof, and/or other typesof neural network architectures.

The processor 404 is configured to send the output data 474 to at leastone of the transmission medium 418 or the storage medium 414. Forexample, the output data 474 may be stored at the storage medium 414 forlater retrieval and decoding (or decompression) by the decoder portion466 to generate the representation 476 of the input data 470 asreconstructed data. The reconstructed data can be used for variouspurposes, such as for playback of video data that has beenencoded/compressed to generate the output data 474. In someimplementations, the output data 474 may be decoded at another decoderdevice that matches the decoder portion 466 (e.g., in the device 402, inthe second device 490, or in another device) to generate therepresentation 476 of the input data 470 as reconstructed data. Forinstance, the second device 490 may include a decoder that matches (orsubstantially matches) the decoder portion 466, and the output data 474may be transmitted via the transmission medium 418 to the second device490. The second device 490 can process the output data 474 to generatethe representation 476 of the input data 470 as reconstructed data.

The components of the system 400 can include and/or can be implementedusing electronic circuits or other electronic hardware, which caninclude one or more programmable electronic circuits (e.g.,microprocessors, graphics processing units (GPUs), digital signalprocessors (DSPs), central processing units (CPUs), and/or othersuitable electronic circuits), and/or can include and/or be implementedusing computer software, firmware, or any combination thereof, toperform the various operations described herein.

While the system 400 is shown to include certain components, one ofordinary skill will appreciate that the system 400 can include more orfewer components than those shown in FIG. 4 . For example, the system400 can also include, or can be part of a computing device thatincludes, an input device and an output device (not shown). In someimplementations, the system 400 may also include, or can be part of acomputing device that includes, one or more memory devices (e.g., one ormore random access memory (RAM) components, read-only memory (ROM)components, cache memory components, buffer components, databasecomponents, and/or other memory devices), one or more processing devices(e.g., one or more CPUs, GPUs, and/or other processing devices) incommunication with and/or electrically connected to the one or morememory devices, one or more wireless interfaces (e.g., including one ormore transceivers and a baseband processor for each wireless interface)for performing wireless communications, one or more wired interfaces(e.g., a serial interface such as a universal serial bus (USB) input,and/or other wired interface) for performing communications over one ormore hardwired connections, and/or other components that are not shownin FIG. 4 .

In some implementations, the system 400 can be implemented locally byand/or included in a computing device. For example, the computing devicecan include a mobile device, a personal computer, a tablet computer, avirtual reality (VR) device (e.g., a head-mounted display (HMD) or otherVR device), an augmented reality (AR) device (e.g., an HMD, AR glasses,or other AR device), a wearable device, a server (e.g., in a software asa service (SaaS) system or other server-based system), a television,and/or any other computing device with the resource capabilities toperform the techniques described herein. In one example, the E2E-NNVCsystem 410 can be incorporated into a portable electronic device thatincludes the memory 406 coupled to the processor 404 and configured tostore instructions executable by the processor 404, and a wirelesstransceiver coupled to an antenna and to the processor 404 and operableto transmit the output data 474 to a remote device.

FIG. 5 is a diagram illustrating examples of different types of motionestimations that can be performed to determine motion informationbetween reference frames (e.g., from a reference frame {circumflex over(X)}_(ref) ₀ to a reference frame {circumflex over (X)}_(ref) ₁ or viceversa). In FIG. 5 , the term x denotes a reference frame from whichmotion can be estimated, the term f denotes a motion estimation, and theterm y denotes a warped frame that can be computed as follows: y=f(x).One type of motion estimation is a block-based motion estimationtechnique 502. The block-based motion estimation can be performed on ablock-by-block basis. For instance, for each block in the frame y, themotion estimation f defines the location of the corresponding block inthe frame x. In one illustrative example, the motion estimation f caninclude a motion vector that indicates the displacement (e.g., thehorizontal and vertical displacement) of a block in the frame y relativeto the corresponding block in the frame x. A block from the frame x canbe determined to correspond to a block in the frame y by determining asimilarity (e.g., a similarity in pixel values) between the blocks.

Another type of motion estimation that can performed is an optical flowmotion estimation technique 504. The optical flow motion estimation canbe performed on a pixel-by-pixel basis. For instance, for each pixel inthe frame y, the motion estimation f defines the location of thecorresponding pixel in the frame x. The motion estimation f for eachpixel can include motion information such as a vector (e.g., a motionvector) indicating a movement of the pixel between the frames. In somecases, optical flow maps (e.g., also referred to as motion vector maps)can be generated based on the computation of the optical flow vectorsbetween frames. The optical flow maps can include an optical flow vectorfor each pixel in a frame, where each vector indicates a movement of apixel between the frames. In one illustrative example, the optical flowvector for a pixel can be a displacement vector (e.g., indicatinghorizontal and vertical displacements, such as x- and y-displacements)showing the movement of a pixel from a first frame to a second frame.

In some cases, the optical flow map can include vectors for less thanall pixels in a frame. For instance, a dense optical flow can becomputed between frames to generate optical flow vectors for each pixelin a frame, which can be included in a dense optical flow map. In someexamples, each optical flow map can include a 2D vector field, with eachvector being a displacement vector showing the movement of points from afirst frame to a second frame.

As noted above, an optical flow vector or optical flow map can becomputed between frames of a sequence of frames. Two frames can includetwo directly adjacent frames that are consecutively captured frames ortwo frames that are a certain distance apart (e.g., within two frames ofone another, within three frames of one another, or any other suitabledistance) in a sequence of frames. In one illustrative example, a pixelI(x,y,t) in the frame x can move by a distance or displacement (Δx,Δy)in the frame y.

In some cases, machine learning-based video coding systems (e.g., suchas neural video systems or codecs) can be used to encode and/or decodevideo data that includes frames of rendered content. For example, therendered content can be generated by cloud gaming systems and/orservers, AR devices, VR devices, etc. In some aspects, rendered contentcan be generated based on one or more user inputs, as will be describedin greater depth below.

FIG. 6 is a diagram illustrating an example of a neural video codingsystem 600 that can be used to encode and/or decode rendered contentassociated with a cloud gaming system. A cloud gaming system can includeone or more servers that run video games locally (e.g., on the server)and stream the output video frames to a user device that is remote fromthe cloud gaming server. By running video games on remote (e.g., cloud)servers or other suitable hardware processing devices, users can playvideo games on smaller and more portable devices, such as smartphones,that otherwise may not have sufficient computational power to run thevideo games locally.

For example, as depicted in FIG. 6 , a cloud gaming server can include agame engine that is used to run the video game or otherwise perform theunderlying processing, calculation, etc., associated with running thevideo game. The game engine can receive one or more user input commandsfor the currently running video game. For example, the user inputcommands can be obtained at a client device, such as a user'ssmartphone, or other user device connected to the cloud gaming serverfor a cloud gaming session. In some examples, the user input commandscan include movement or directional inputs (e.g., using a joystick orhardware controller coupled to the client device, using on-screen orsoftware controller elements provided by the client device, etc.),although it is noted that various other user input commands may also bereceived and utilized by the cloud gaming server.

Based on the user input commands, the game engine included in the cloudgaming server can calculate the effect or result of the user inputcommands in the video game environment and may update the current gamestate accordingly. In one illustrative example, the game engine cangenerate one or more updated frames of rendered video data based on theuser input commands and the change(s) to the current game state. The oneor more updated frames of rendered video data can be written to orotherwise stored in one or more buffers included in the cloud gamingserver.

In some cases, the one or more buffers can include a deferred shadingbuffer and/or a geometry buffer (G-buffer). For example, a G-buffer canbe used to store screen space representations of geometry and materialinformation (e.g., generated by an intermediate rendering pass in adeferred shading rendering pipeline associated with the game engine). Insome examples, the G-buffer can include rendering information associatedwith one or more frames of the cloud gaming video data. The renderinginformation can include, but is not limited to, motion information,depth information, normal information, albedo information, etc. In somecases, the rendering information can be stored in the G-buffer as maps.For example, the rendering information associated with the frames ofcloud gaming video data can include, but is not limited to, motion maps(e.g., optical flow maps), depth maps, normal maps, albedo maps, etc.

The frames of cloud gaming video data can be encoded using a neuralvideo encoder included in the cloud gaming server. As illustrated inFIG. 6 , the neural video encoder can obtain the frames of cloud gamingvideo data (e.g., generated by the game engine in response to the userinput commands) from the G-buffer, and encodes each frame into abitstream.

The bitstream associated with the encoded frames of cloud gaming videodata can be transmitted to the client or user device, where thebitstream is decoded by a neural video decoder. In some examples, thereconstructed frames of cloud gaming video data (e.g., decoded from thebitstream by the client-device neural video decoder) can be provided toa renderer included in the client device, wherein the renderersubsequently renders and displays the reconstructed frames to the user.Based on the display of the reconstructed frames, the user perceivestemporal flow within the video game and provides subsequent user inputsbased on the updated game state—the subsequent user inputs aretransmitted to the game engine of the cloud gaming server, and theexample process described above can be repeated.

Latency can be an important performance factor for cloud gaming systems.For example, an increased latency may be perceived as lag by a user. Insome cases, a latency of 100 milliseconds (ms) or less may be needed forfast-paced video games and a latency of 150 ms or less may be needed forslower-paced games. In some cases, the latency threshold associated withcoding video frames of rendered content, such as cloud gaming content,can be lower than a latency threshold associated with streaming livevideo from a server to a remote client device. For example, live videostreams may tolerate a latency of up to 500 ms, based at least in parton live video streams being primarily uni-directional. The maximumlatency threshold associated with coding video frames of renderedcontent can be the maximum end-to-end latency that may be tolerated byusers, and can be impacted by the speed of the game engine, the speed ofthe server-side neural video encoder, the speed of the communicationdownlink from the cloud gaming server to the client device (e.g., usedto transmit the encoded bitstream), the speed of the neural videodecoder, and/or the speed of the communication uplink from the clientdevice to the cloud gaming server (e.g., used to transmit the user inputcommands), etc.

In some examples, frames of rendered content (e.g., such as cloud gamingcontent) can be encoded and decoded based on or using a neuralnetwork-based prediction frame (P-frame) coding system, wherein acurrent frame is encoded/decoded based on information determined forprevious frames (e.g., determined in previous time steps). In someaspects, frames of rendered content can be encoded and decoded based onor using a neural network-based P-frame coding system such as theexample neural P-frame coding system 700 depicted in FIG. 7 . Asillustrated, the example neural P-frame coding system 700 includes amotion prediction system 702, a warping engine 704, and a residualprediction system 706. The motion prediction system 702 and the residualprediction system 706 can include any type of machine learning system(e.g., using one or more neural networks and/or other machine learningmodels, architectures, networks, etc.).

In some aspects, the motion prediction system 702 can include one ormore autoencoders. In one illustrative example, the encoder network 705and the decoder network 703 of motion prediction system 702 can beimplemented as an autoencoder (e.g., also referred to as a “motionautoencoder” or “motion AE”). In some cases, the motion predictionsystem 702 can be used to implement an optical flow-based motionestimation techniques, such as the example optical flow motionestimation technique 504 illustrated in FIG. 5 , and the motionautoencoder can be implemented as an optical flow autoencoder (e.g.,also referred to as a “flow autoencoder” or “flow AE”). In some aspects,the residual prediction system 706 can include one or more autoencoders.In one illustrative example, the encoder network 707 and the decodernetwork 709 of residual prediction system 706 can be implemented as anautoencoder (e.g., also referred to as a “residual autoencoder” or“residual AE”). While the example P-frame coding system 700 of FIG. 7 isshown to include certain components, one of ordinary skill willappreciate that the example P-frame coding system 700 can include feweror more components than those shown in FIG. 7 .

In one illustrative example, for a given time t, the neural P-framecoding system 700 can receive an input frame X_(t) and a reference frame{circumflex over (X)}_(t-1). In some aspects, the reference frame{circumflex over (X)}_(t-1) can be a previously reconstructed frame(e.g., as indicated by the hat operator “i”) generated prior to time t(e.g., at time t−1). Input frame X_(t) and reference frame {circumflexover (X)}_(t-1) can be associated with or otherwise obtained from thesame sequence of video data (e.g., as consecutive frames, etc.). Forexample, the input frame X_(t) can be the current frame at time t, andthe reference frame {circumflex over (X)}_(t-1) can be a frametemporally or sequentially immediately prior to the input frame X_(t).In some cases, the reference frame {circumflex over (X)}_(t-1) may bereceived from a decoded picture buffer (DPB) of the example neuralP-frame coding system 700. In some cases, the input frame X_(t) can be aP-frame and the reference frame {circumflex over (X)}_(t-1) can be anI-frame, a P-frame, or a B-frame. For example, the reference frame{circumflex over (X)}_(t-1) can be previously reconstructed or generatedby an I-frame coding system (e.g., which can be part of a device whichincludes the P-frame coding system 700 or a different device than thatwhich includes the P-frame coding system 700), by the P-frame codingsystem 700 (or a P-frame coding system of a device other than that whichincludes the P-frame coding system 700), or by a B-frame coding system(e.g., which can be part of a device which includes the P-frame codingsystem 700 or a different device than that which includes the P-framecoding system 700).

As depicted in FIG. 7 , motion prediction system 702 (e.g., a motionautoencoder, optical flow autoencoder, etc.) receives as input referenceframe {circumflex over (X)}_(t-1) and the current (e.g., input) frameX_(t). Motion prediction system 702 can determine motion (e.g.,represented by vectors, such as optical flow motion vectors) betweenpixels of reference frame {circumflex over (X)}_(t-1) and pixels ofinput frame X_(t). Motion prediction system 702 can then encode, and insome cases decode, this determined motion as a predicted motion{circumflex over (f)}_(t) for input frame X_(t).

For example, an encoder network 705 of motion prediction system 702 canbe used to determine motion (e.g., motion information) between currentframe X_(t) and reference frame {circumflex over (X)}_(t-1). In someaspects, encoder network 705 can encode the determined motioninformation into a latent representation (e.g., denoted as latentz_(m)). For example, in some cases encoder network 705 can map thedetermined motion information to a latent code, which can be used as thelatent z_(m). Encoder network 705 can additionally, or alternatively,convert the latent z_(m) into a bitstream by performing entropy codingon the latent code associated with z_(m). In some examples, encodernetwork 705 can quantize the latent z_(m) (e.g., prior to entropy codingbeing performed on the latent code). The quantized latent can include aquantized representation of the latent z_(m). In some cases, the latentz_(m) can include neural network data (e.g., a neural network node'sactivation map or feature map) that represents one or more quantizedcodes.

In some aspects, encoder network 705 can store the latent z_(m), sendthe latent z_(m) to a decoder network 703 included in motion predictionsystem 702, and/or can send the latent z_(m) to another device or systemthat can decode the latent z_(m). Upon receiving the latent z_(m),decoder network 703 can decode (e.g., inverse entropy code, dequantize,and/or reconstruct) the latent z_(m) to generate a predicted motion{circumflex over (f)}_(t) between pixels of reference frame {circumflexover (X)}_(t-1) and pixels of input frame X_(t). For example, decodernetwork 703 can decode the latent z_(m) to generate an optical flow map{circumflex over (f)}_(t) that includes one or more motion vectorsmapping some (or all) of the pixels included in reference frame{circumflex over (X)}_(t-1) to pixels of input frame X_(t). Encodernetwork 705 and decoder network 703 can be trained and optimized usingtraining data (e.g., training images or frames) and one or more lossfunctions, as will be described in greater depth below.

In one illustrative example, encoder network 705 and decoder network 703can be included in a motion prediction autoencoder, such as an opticalflow autoencoder. The optical flow autoencoder can include one or morecomponents for quantizing the latent z_(m) (e.g., generated as output byencoder network 705 of the optical flow autoencoder) and converting thequantized latent into a bitstream. The bitstream generated from thequantized latent can be provided as input to decoder 703 of the opticalflow autoencoder.

In some examples, predicted motion {circumflex over (f)}_(t) can includeoptical flow information or data (e.g., an optical flow map includingone or more motion vectors), dynamic convolution data (e.g., a matrix orkernel for data convolution), or block-based motion data (e.g., a motionvector for each block). In one illustrative example, predicted motion{circumflex over (f)}_(t) can include an optical flow map. In somecases, as described previously, an optical flow map {circumflex over(f)}_(t) can include a motion vector for each pixel of input frame X_(t)(e.g., a first motion vector for a first pixel, a second motion vectorfor a second pixel, and so on). The motion vectors can represent themotion information determined (e.g., by encoder network 705) for thepixels in current frame X_(t) relative to corresponding pixels inreference frame {circumflex over (X)}_(t-1).

The warping engine 704 of neural P-frame coding system 700 can obtainthe optical flow map {circumflex over (f)}_(t) generated as output bymotion prediction system 702 (e.g., generated as output by decodernetwork 703/the optical flow autoencoder described above). For example,warping engine 704 can retrieve optical flow map {circumflex over(f)}_(t) from storage or can receive optical flow map {circumflex over(f)}_(t) from motion prediction system 702 directly. Warping engine 704can use optical flow map {circumflex over (f)}_(t) to warp (e.g., byperforming motion compensation) the pixels of reference frame{circumflex over (X)}_(t-1), resulting in the generation of a warpedframe {tilde over (X)}_(t). In some aspects, warped frame {tilde over(X)}_(t) can also be referred to as a motion compensated frame {tildeover (X)}_(t) (e.g., generated by warping the pixels of reference frame{circumflex over (X)}_(t-1) based on the corresponding motion vectorsincluded in optical flow map {circumflex over (f)}_(t)). For example,warping engine 704 can generate motion compensated frame {tilde over(X)}_(t) by moving the pixels of reference frame {circumflex over(X)}_(t-1) to new locations based on the motion vectors (and/or othermotion information) included in optical flow map {circumflex over(f)}_(t).

As noted above, to generate warped frame {tilde over (X)}_(t), neuralP-frame coding system 700 can perform motion compensation by predictingan optical flow {circumflex over (f)}_(t) between input frame X_(t) andreference frame {circumflex over (X)}_(t-1), and subsequently generatinga motion compensated frame {tilde over (X)}_(t) by warping referenceframe {circumflex over (X)}_(t-1) using the optical flow map {circumflexover (f)}_(t). However, in some cases, the frame prediction (e.g.,motion-compensated frame {tilde over (X)}_(t)) generated based onoptical flow map {circumflex over (f)}_(t) may not be accurate enough torepresent input frame X_(t) as a reconstructed frame {circumflex over(X)}_(t). For example, there may be one or more occluded areas in ascene depicted by input frame X_(t), excessive lighting, lack oflighting, and/or other effects that results in the motion-compensatedframe {tilde over (X)}_(t) not being accurate enough to for use as areconstructed input frame {circumflex over (X)}_(t).

Residual prediction system 706 can be used to correct or otherwiserefine the prediction associated with motion-compensated frame {tildeover (X)}_(t). For example, residual prediction system 706 can generateone or more residuals that neural P-frame coding system 700 cansubsequently combine with motion-compensated frame {tilde over (X)}_(t)in order to thereby generate a more accurate reconstructed input frame{circumflex over (X)}_(t) (e.g., a reconstructed input frame {circumflexover (X)}_(t) that more accurately represents the underlying input frameX_(t)). In one illustrative example, as depicted in FIG. 7 , neuralP-frame coding system 700 can determine a residual r_(t) by subtractingthe predicted (e.g., motion-compensated) frame {tilde over (X)}_(t) frominput frame X_(t) (e.g., determined using a subtraction operation 708).For example, after the motion-compensated predicted frame {tilde over(X)}_(t) is determined by warping engine 704, P-frame coding system 700can determine the residual r_(t) by determining the difference (e.g.,using subtraction operation 708) between motion-compensated predictedframe {tilde over (X)}_(t) and input frame X_(t).

In some aspects, an encoder network 707 of residual prediction system706 can encode the residual r_(t) into a latent z_(r), where the latentz_(r) represents the residual r_(t). For example, encoder network 707can map the residual r_(t) to a latent code, which can be used as thelatent z_(r). In some cases, encoder network 707 can convert the latentz_(r) into a bitstream by performing entropy coding on the latent code.In some examples, encoder network 707 can additionally, oralternatively, quantize the latent z_(r) (e.g., before entropy coding isperformed). The quantized latent z_(r) can include a quantizedrepresentation of the residual r_(t). In some cases, the latent z_(r)can include neural network data (e.g., a neural network node'sactivation map or feature map) that represents one or more quantizedcodes. In some aspects, encoder network 707 can store the latent z_(r),transmit or otherwise provide the latent z_(r) to a decoder network 709of residual prediction system 706, and/or can send the latent z_(r) toanother device or system that can decode the latent z_(r). Uponreceiving the latent z_(r), decoder network 709 can decode the latentz_(r) (e.g., inverse entropy code, dequantize, and/or reconstruct) togenerate a predicted (e.g., decoded) residual {circumflex over (r)}_(t).In some examples, encoder network 707 and decoder network 709 can betrained and optimized using training data (e.g., training images orframes) and one or more loss functions, as described below.

In one illustrative example, encoder network 707 and decoder network 709can be included in a residual prediction autoencoder. The residualautoencoder can include one or more components for quantizing the latentz_(r) (e.g., where the latent z_(r) is generated as output by encodernetwork 707 of the residual autoencoder) and converting the quantizedlatent into a bitstream. The bitstream generated from the quantizedlatent z_(r) can be provided as input to decoder 709 of the residualautoencoder.

The predicted residual {circumflex over (r)}_(t) (e.g., generated bydecoder network 709 and/or a residual autoencoder used to implementresidual prediction system 706) can be used with the motion-compensatedpredicted frame {tilde over (X)}_(t) (e.g., generated by warping engine704 using the optical flow map {circumflex over (f)}_(t) generated bydecoder network 703 and/or an optical flow autoencoder used to implementmotion prediction system 702) to generate a reconstructed input frame{circumflex over (X)}_(t) representing the input frame X_(t) at time t.

For example, neural P-frame coding system 700 can add (e.g., usingaddition operation 710) or otherwise combine the predicted residual{circumflex over (r)}_(t) and the motion-compensated predicted frame{tilde over (X)}_(t) to generate the reconstructed input frame{circumflex over (X)}_(t). In some cases, decoder network 709 ofresidual prediction system 706 can add the predicted residual{circumflex over (r)}_(t) to the motion-compensated frame prediction{tilde over (X)}_(t). In some examples, reconstructed input frame{circumflex over (X)}_(t) may also be referred to as a decoded frameand/or a reconstructed current frame. The reconstructed current frame{circumflex over (X)}_(t) can be output for storage (e.g., in a decodedpicture buffer (DPB) or other storage), transmission, display, forfurther processing (e.g., as a reference frame in furtherinter-predictions, for post-processing, etc.), and/or for any other use.

In one illustrative example, P-frame coding system 700 can transmit thelatent data representing an optical flow map or other motion information(e.g., the latent z_(m)) and the latent data representing the residualinformation (e.g., the latent z_(r)) in one or more bitstreams toanother device for decoding. In some cases, the other device can includea video decoder configured to decode the latents z_(m) and z_(r). In oneillustrative example, the other device can include a video decoderimplementing one or more portions of P-frame coding system 700, motionprediction system 702 (e.g., an optical flow autoencoder), and/orresidual prediction system 706 (e.g., a residual autoencoder), asdescribed above.

The other device or video decoder can decode the optical flow map{circumflex over (f)}_(t) and/or other predicted motion informationusing the latent z_(m) generated as output by the decoder network 703included in motion prediction system 702 (e.g., generated as output byan optical flow autoencoder that includes decoder network 703). Theother device or video decoder can additionally decode the residual{circumflex over (r)}_(t) using the latent z_(r) generated as output bythe decoder network 709 included in residual prediction system 706(e.g., generated as output by a residual autoencoder that includesdecoder network 709). The other device or video decoder can subsequentlyuse the optical flow map {circumflex over (f)}_(t) and the residual{circumflex over (r)}_(t) to generate the decoded (e.g., reconstructed)input frame {circumflex over (X)}_(t).

For example, when the video decoder implements a same or similararchitecture to that of the P-frame coding system 700 described above,the video decoder can include a warping engine (e.g., the same as orsimilar to warping engine 704) that receives as input the decodedoptical flow map {circumflex over (f)}_(t) and the reference frame{circumflex over (X)}_(t-1). The video decoder warping engine can warpreference frame {circumflex over (X)}_(t-1) based on motion vectorsand/or other motion information determined for some (or all) of thepixels of reference frame {circumflex over (X)}_(t-1), based on thedecoded optical flow map {circumflex over (f)}_(t). The video decoderwarping engine can output a motion-compensated frame prediction {tildeover (X)}_(t) (e.g., as described above with respect to the output ofwarping engine 704). The video decoder can subsequently add or otherwisecombine the decoded residual {circumflex over (r)}_(t) and themotion-compensated frame prediction {tilde over (X)}_(t) to generate thedecoded (e.g., reconstructed) input frame {circumflex over (X)}_(t)(e.g., as described above with respect to the output of additionoperation 710).

In some examples, motion prediction system 702 and/or residualprediction system 706 can be trained and/or optimized using trainingdata and one or more loss functions. In some cases, motion predictionsystem 702 and/or residual prediction system 706 can be trained in anend-to-end manner (e.g., where all neural network components are trainedduring the same training process). As mentioned previously, in someexamples motion prediction system 702 can be implemented as an opticalflow autoencoder including encoder network 705 and decoder network 703;and residual prediction system 706 can be implemented as a residualautoencoder including encoder network 707 and decoder network 709. Insome aspects, the training data can include a plurality of trainingimages and/or training frames. In some cases, a loss function (e.g.,Loss) can be used to perform training, based on motion prediction system702 and/or residual prediction system 706 processing the training imagesor frames.

In one example, the loss function (e.g., Loss) can be given asLoss=D+βR, where D is a distortion between a given frame (e.g., such asinput frame X_(t)) and its corresponding reconstructed frame (e.g.,{circumflex over (X)}_(t)). For example, the distortion D can bedetermined as D(X_(t), {circumflex over (X)}_(t)). β is a hyperparameterthat can be used to control a bitrate (e.g., bits per pixel), and R is aquantity of bits used to convert the residual (e.g., residual r_(t)) toa compressed bitstream (e.g., latent z_(r)). In some examples, thedistortion D can be calculated based on one or more of a peaksignal-to-noise ratio (PSNR), a structural similarity index measure(SSIM), a multiscale SSIM (MS-SSIM), and/or the like. In some aspects,using one or more training data sets and one or more loss functions,parameters (e.g., weights, biases, etc.) of motion prediction system 702and/or residual prediction system 706 can be tuned until a desired videocoding result is achieved by example neural P-frame coding system 700.

In some examples, one or more of the encoder network(s), decodernetwork(s), autoencoder network(s), and/or codec(s) described herein canbe implemented based on or using a Scale-Space Flow (SSF) architecture.In some aspects, an SSF architecture can be augmented with one or motioncompensation steps. For example, augmenting an SSF architecture with atleast one motion compensation step can make better use of availablerendered content (e.g., one or more depth maps, one or more optical flowmaps, etc.). In some examples, an SSF architecture can be augmented withone or more motion compensation steps to correct for (e.g., compensatefor) camera motion. Correcting or compensating for camera motion canimprove rate distortion and coding efficiency of the augmented SSFarchitecture and/or one or more of the codecs implemented using thesystems and techniques described herein. In some aspects, region ofinterest-coding (ROI-coding) can be implemented based on one or moreROI-aware losses and/or the conditioning of one or more encoder networkson a ROI mask.

In some examples, the systems and techniques described herein canperform or apply camera motion compensation for inputs that includerendered content. In some aspects, the systems and techniques describedherein can be associated with a Bjøntegaard Delta (BD) rate savings onthe TartanAir dataset (e.g., over an SSF architecture analyzed on thesame TartanAir dataset). In some cases, the use of ROI-coding may beassociated with greater coding flexibility for the systems andtechniques described herein. For example, one or more depth maps can beused as a proxy for one or more ROI maps. In some aspects, the use ofROI-coding may be associated with a rate savings compared to a non-ROIimplementation (e.g., SSF), while maintaining perceived quality of thecoded input(s) and/or output(s). In some examples, one or more GameCodecparameters can be adapted to specific game environments, and may beassociated with an additional BD rate gain. In some cases, the BD rategain associated with adapting GameCodec parameters to gamer environmentscan be based at least in part on a scene type associated with a givengame environment. In some examples, the systems and techniques can beused to implement instance-adaptive coding.

In some examples, an SSF-based implementation can be trained using therate-distortion (RD) loss

, with a parameter β used to tradeoff rate for distortion (e.g.,

=D+β·R). In some aspects, the systems and techniques can performtraining based on a modified ROI-aware objective, for example using aloss function

_(ROI)=D_(ROI)+βR.

D_(ROI) is the Mean-Squared Error distortion metric between aground-truth frame sequence x and its reconstruction {circumflex over(x)}, weighted by the importance map sequence m whose values are in(0,1) (e.g., with N representing the total number of elements in thesequence):

$D_{ROI} = {\frac{1}{N}{\sum}_{i}^{N}{m_{i} \cdot \left( {x_{i} -} \right)^{2}}}$

In some examples,

_(ROI) can be a generalized loss definition, supporting the case wherethe importance mask is continuous instead of binary (e.g., based on theuse of depth map(s) as surrogate(s) for importance, together with thehuman visual system model of acuity decreasing exponentially from afocal point and leading to continuous importance maps with peaks nearestimated focal points and close objects).

In some aspects, training can be performed based on sampling atemporally-coherent Perlin noise mask m_(p) of same dimension as theframe sequence x. The noise mask m_(p) can be rescaled to (0,1) and usedto define the importance mask as m=1−α·m_(p), where α is a scalar in(0,1) which controls the focus on ROI. In some cases, if α is 0, thenthe importance map m may be all-ones, and

_(ROI) degenerates to the non-ROI case

.

In some examples, an evaluation phase of training can use an importancemap based on the depth mask and the hyperparameter a, which can be usedto control or adjust the amount of disparity that should be presentbetween foreground and background regions in terms of image quality. Forexample, denoting the depth map sequence z, an importance map m can beused where

${m = {1 - {\alpha \cdot \frac{z - {z_{\min}\gamma}}{z_{\max} - z_{\min}}}}},$

where z_(min) and z_(max) are the minimum and maximum value(respectively) for depth for the associated game engine, and γ is ahyperparameter controlling the power-law transformation of the depthmap. In some cases, γ can be adjusted at test time to control thedrop-off between foreground and background (e.g., softer if γ<1 andsteeper if γ>1).

In some aspects, learned codecs can be more easily finetuned to anapplication domain (e.g., relative to standard codecs). In the exampleof cloud gaming, the systems and techniques can use one or more models,neural network, machine learning networks, autoencoders, etc., that arefinetuned on different game scenarios in a dataset. A boost in R-Dperformance can be observed when evaluating on unseen sequences from thesame game. In some aspects, updated model weights can be transmitted toupdate a receiver-side codec prior to starting a transmission (e.g.,representing at most tens of megabytes).

As mentioned previously, in some examples frames of rendered videocontent (e.g., such as cloud gaming content) can be encoded and decodedusing a neural video coding system that is the same as or similar to theexample neural P-frame coding system 700 described above. For example, acurrent frame of cloud gaming content (or other rendered video content)can be coded based on determining an optical flow map or other motioninformation for the current frame, along with a residual map.

In some aspects, the motion between successive frames of video contentcan be based on object motion and camera motion. In the example of FIG.7 , both object motion and camera motion are predicted or otherwisedetermined using the single motion prediction system 702. In someexamples, the use of a single motion prediction system to performcamera-based motion compensation and object-based motion compensationcan decrease the performance and/or accuracy of the example neuralP-frame coding system 700, such as for coding video content thatincludes significant camera motion over short periods of time. Forexample, frames of cloud gaming content may include significant cameramotion over short periods of time (e.g., based on rapid user inputs,point-of-view manipulations, etc., while gaming). In some aspects, thesystems and techniques described herein can be used to encode and decodeframes of video content, including frames of rendered video content,based on decomposing camera motion compensation and object motioncompensation, as will be described in greater depth below.

FIG. 8 is a diagram illustrating an example of a machine learning-basedvideo coding system 800 that includes a depth map prediction system 822and an object motion prediction system 802. In one illustrative example,depth map prediction system 822 can be used to compensate camera motionusing rendering information associated with the currently encoded ordecoded frame of rendered video data (e.g., a frame of rendered cloudgaming content), as will be described in greater depth below. In someaspects, object motion prediction system 802 can be used to compensateobject motion using optical flow-based warping (e.g., as described abovewith respect to object motion prediction system 702 illustrated in FIG.7 ). In some cases, object motion prediction system 802 can be the sameas or similar to the object motion prediction system 702 illustrated inFIG. 7 . For example, object motion prediction system 802 can beimplemented as an optical flow autoencoder that includes the encodernetwork 805 and the decoder network 803.

As described previously, video content that includes rendered frames ofvideo data (e.g., computer generated or animated visual content, ARcontent, VR content, gaming content, etc.) may include relativelygreater motion between successive frames than natural video content(e.g., video content captured by a camera). In some cases, renderedframes of video data can include relatively greater camera motion andrelatively greater object motion than a natural content video data.

In the example of a rendered frames of gaming content (e.g., generatedby or otherwise obtained from a cloud gaming system such as the cloudgaming system 600 illustrated in FIG. 6 ), an associated camera motionbetween successive frames can be relatively high based on frequentand/or large movements made by a user playing the video game, whereinthe user movements or inputs are associated with corresponding changesin the camera point-of-view (POV) and/or field-of-view (FOV). Also inthe example of rendered frames of gaming content, an associated objectmotion between successive frames can be relatively high based on factorssuch as the rendered frames of gaming content corresponding to largeand/or complicated scenes (e.g., which may include large-sized objects,highly detailed or granular objects, various visual and/or particleeffects such as light, rain, smoke, etc.)

In one illustrative example, the systems and techniques described hereincan perform video coding for frames of video content (e.g., frames ofrendered and/or gaming content) based on using separate motionprediction systems to perform camera motion compensation and to performobject motion compensation. For example, depth map prediction system 822can be used to perform camera motion compensation based on renderinginformation associated with one or more frames of the video content, aswill be explained in greater depth below.

In one illustrative example, rendering information associated with oneor more frames of the currently encoded/decoded video content caninclude a current camera pose p_(t) (e.g., the camera pose associatedwith the current frame t), a reference camera pose p_(t-1) (e.g., thecamera pose associated with the prior frame t−1), and a depth map D_(t)associated with the current frame t. Camera pose information can includeposition and orientation information of a camera with respect to areference coordinate system of the scene depicted in the video content(e.g., the scene captured by the camera). A depth map can be an imagethat includes information associated with the distance of the surfacesof scene objects from a given viewpoint (e.g., the viewpoint of thecamera that captures the scene). In some examples, a depth map can havethe same pixel dimensions as a corresponding frame of video data forwhich the depth map is generated, wherein some (or all) of the pixelsincluded in the depth map are associated with a depth value. The depthvalue associated with a given pixel can indicate a distance from thecamera viewpoint to the surface of the scene object depicted by thegiven pixel.

In examples in which the frames of video content coded (e.g., encodedand/or decoded) using the example neural video coding system 800 areframes of rendered video data, the camera pose information and depthinformation may be generated in association with rendering the videoframes (e.g., generated at the time the video frame is originallygenerated, prior to encoding or decoding). For example, camera pose anddepth information can be obtained from one or more rendering enginesassociated with generating the rendered frames, based on the renderingengines having used the camera pose and depth information to originallygenerate the rendered frames. In some aspects, camera pose and depthinformation associated with frames of rendered gaming content can beobtained from one or more buffers associated with a rendering or gameengine used to generate the gaming content. For example, camera pose anddepth information associated with frames of rendered gaming contentgenerated by the game engine depicted in the cloud gaming server of FIG.6 can be obtained from a deferred rendering buffer and/or geometrybuffer (G-buffer) associated with the game engine. In some examples, thecamera pose and depth information may be stored in a deferred renderingbuffer or G-buffer among various other types of rendering information(e.g., motion information, optical flow information such as optical flowmaps, normal maps, albedo maps, etc.).

In some examples, the systems and techniques described herein can beused to perform video coding based on separate camera motioncompensation and object motion compensation for non-rendered videocontent (e.g., natural video content). For example, the camera poseinformation p_(t) and p_(t-1) can be obtained from a camera used tocaptured the non-rendered or natural video content, can be determinedusing one or more post-processing algorithms or machine-learningnetworks, etc. In some cases, depth information can be obtained from acamera that includes one or more depth sensors and captures depthinformation and video content simultaneously (e.g., such that the videocontent is captured in combination with a corresponding depth map forone or more frames of the captured video content). In some aspects,depth information can be obtained from a stereo camera that includes twoor more cameras for determining depth maps.

In one illustrative example, depth map prediction system 822 and 3Dwarping engine 824 can be used to generate an initial frame prediction X_(t) that is compensated using camera motion determined based on theinput camera pose p_(t), reference camera pose p_(t-1), and depth mapD_(t). In some aspects, the initial frame prediction X _(t) may also bereferred to as a camera motion compensated frame. As illustrated, 3Dwarping engine 824 can generate the initial frame prediction X _(t)based on receiving as input the camera pose p_(t) for the current framet, the reference camera pose p_(t-1) for the previous frame t−1, thepreviously decoded (e.g. reconstructed) frame {circumflex over(X)}_(t-1), and a predicted (e.g., reconstructed) depth map {circumflexover (d)}_(t) for the current frame t.

The camera poses p_(t) and p_(t-1), along with the depth map D_(t) canbe generated by a rendering engine (e.g., a cloud gaming engine)associated with a neural video encoder implementing the examplearchitecture of FIG. 8 . In one illustrative example, the neural videoencoder implementing the example architecture of FIG. 8 can be includedin a same server or other computing device as the rendering engine usedto generate the frames of rendered video data being encoded (e.g., whichmay also be the same rendering engine used to generate the camera posesp_(t) and p_(t-1), along with the depth map D_(t)).

A neural video decoder implementing some or all of the examplearchitecture of FIG. 8 can receive the camera poses p_(t) and p_(t-1)from the neural video encoder and/or the server (e.g., rendering engine)associated with the neural video encoder. In some examples, the neuralvideo decoder may receive only the current camera poses p_(t) associatedwith the currently coded frame of rendered video data, based on theneural video decoder receiving the reference camera pose p_(t-1) as thecurrent camera pose for the prior frame (e.g., the neural video decodercan store the camera poses received from the neural video encoder foruse in decoding future or subsequent frames of the rendered videocontent).

In some aspects, rather than transmitting the depth map D_(t) from theneural video encoder to the neural video decoder, the systems andtechniques can use depth map prediction system 822 to perform depth mapcompression. For example, depth map D_(t) can be provided as input to anencoder network 825 included in depth map prediction system 822, whichin some cases may be implemented as an autoencoder. Encoder network 825can generate as output a latent representation z_(c) of the depth mapD_(t). In some aspects, the latent representation z_(c) can be convertedinto a bitstream and transmitted to the neural video decoderimplementing the example architecture of FIG. 8 .

For example, the latent representation z_(c) of depth map D_(t) can beprovided as input to a decoder network 823 included in depth mapprediction system 822. In some aspects, both the neural video encoderand the neural video decoder can include the decoder network 823. Basedon receiving the latent representation z_(c) as input, the decodernetwork 823 can generate as output a predicted depth map {circumflexover (d)}_(t) for the currently coded frame t. In some examples, thepredicted depth map {circumflex over (d)}_(t) may also be referred to asthe “reconstructed depth map.”

3D warping engine 824 can generate the initial frame prediction X _(t)(e.g., the camera motion compensated frame) by warping the previouslyreconstructed frame {circumflex over (X)}_(t-1) based on the cameraposes p_(t) and p_(t-1), and the reconstructed depth map {circumflexover (d)}_(t). For example, the camera poses p_(t) and p_(t-1) can beused to estimate a 3D camera transformation, T(p_(t), p_(t-1)),associated with the motion or movement of the camera between the pose attime t and the pose at time t−1. An inverse projection can be performedbased on the reconstructed depth map {circumflex over (d)}_(t) and acamera projection matrix P (e.g., also referred to as a “backprojection”). The output of the inverse projection can be a set of 3Dscene coordinates with the camera at time t.

The estimated 3D camera transformation T(p_(t), p_(t-1)) and the outputof the inverse projection (e.g., the set of 3D scene coordinates withthe camera at time t) can be provided as inputs to a 3D pointtransformation, which generates as output a set of 3D scene coordinateswith the camera at time t−1. A camera projection can be performed totransform the 3D scene coordinates with the camera at time t−1 into aset of 2D scene coordinates with the camera at time t−1. The cameraprojection into the 2D scene coordinates can utilize the same cameraprojection matrix P that was used to perform the prior inverseprojection.

Based on the camera projection of 2D scene coordinates with the cameraat time t−1, an estimated correspondence can be determined between I_(t)(e.g., the frame or scene captured by the camera at time t) and I_(t-1)(e.g., the frame or scene captured by the camera at time t−1). Theestimated 2D correspondence between I_(t) and I_(t-1) can be provided asinput to a bilinear warping engine. In some aspects, the bilinearwarping engine can be the same as or included in the 2D warping engine824. In addition to the estimated 2D correspondence between I_(t) andI_(t-1), the bilinear warping engine can additionally receive as inputthe previously reconstructed frame {circumflex over (X)}_(t-1) (e.g., asillustrated in FIG. 8 with respect to 3D warping engine 824). Based onperforming a bilinear warping between the estimated 2D correspondencebetween I_(t) and I_(t-1), and the previously reconstructed frame{circumflex over (X)}_(t-1), a camera compensated frame can be generatedas output. In some aspects, the camera compensated frame generated asoutput based on the bilinear warping described above can be the same asthe initial frame prediction X _(t) generated by the 3D warping engine824.

The camera motion compensated frame X _(t) can subsequently be providedto an additional warping engine 804. As illustrated, additional warpingengine 804 may also receive as input a predicted optical flow map{circumflex over (f)}_(t) ^(obj) (as an example of optical flowinformation) associated with the object motion for the current frame t.In some aspects, the systems and techniques can perform an accumulativewarping based on using warping engine 804 to warp the camera motioncompensated frame X _(t) with the predicted object motion optical flowmap {circumflex over (f)}_(t) ^(obj) (e.g., an accumulative warpingbased on the camera motion compensated frame X _(t) having beengenerated using a previous warping operation performed by 3D warpingengine 824).

In some aspects, the object motion prediction system 802 can be the sameas or similar to the object motion prediction system 702 described abovewith respect to FIG. 7 , and the object motion optical flow map{circumflex over (f)}_(t) ^(obj) can be generated in a same or similarmanner as the optical flow map {circumflex over (f)}_(t) also describedabove with respect to FIG. 7 . For example, the object motion opticalflow map {circumflex over (f)}_(t) ^(obj) can be generated based onproviding the camera motion compensated frame X _(t) and the currentlycoded frame X_(t) as input to an encoder network 805 of object motionprediction system 802, which may be implemented as an optical flowautoencoder. Encoder network 805 can determine an optical flow betweenthe two input frames, and generate as output a latent representationz_(p) of the determined optical flow map. As described above withrespect to depth map prediction system 822, in some aspects the objectmotion prediction system 802 can convert the latent representation z_(p)into a bitstream that is transmitted from the neural video encoder tothe neural video decoder. The bitstream representation of the latentz_(p) can be provided as input to decoder network 803, which generatesas output the predicted (e.g., reconstructed) optical flow map{circumflex over (f)}_(t) ^(obj) representing the optical flow betweenthe camera motion compensated frame X _(t) and the input (e.g.,currently coded) frame X_(t).

In some aspects, the latent representation z_(c) of the depth map D_(t)can be provided as an additional input to object motion predictionsystem 802. For example, object motion prediction system 802 canconcatenate the latent representation z_(c) of the depth map D_(t) withthe latent representation z_(p) of the optical flow map generated byencoder network 805. Based on receiving as input both the latentrepresentation z_(c) of the depth map D_(t) (e.g., associated with thecamera motion) and the latent representation z_(p) of the optical flowmap (e.g., associated with the object motion), in some aspects decodernetwork 803 can generate the reconstructed optical flow map {circumflexover (f)}_(t) ^(obj) to include object motion information withoutincluding camera motion information. For example, the decoder network803 can remove camera motion information from the reconstructed opticalflow map {circumflex over (f)}_(t) ^(obj) based on analyzing the latentrepresentation z_(c) of the depth map D_(t) (e.g., associated with thecamera motion).

Warping engine 804 can generate as output a predicted motion compensatedframe {tilde over (X)}_(t) by warping the camera motion compensatedframe X _(t) based on the reconstructed object motion optical flow map{circumflex over (f)}_(t) ^(obj) (e.g., in a same or similar manner toas described previously with respect to the fame prediction output bywarping engine 704 illustrated in FIG. 7 ). In one illustrative example,a predicted motion compensated frame {tilde over (X)}_(t) includesmotion compensation for camera motion (e.g., based on the first warpperformed by 3D warping engine 824) and includes motion compensation forobject motion (e.g., based on the second warp performed by warpingengine 804).

The predicted motion compensated frame {tilde over (X)}_(t) can berefined based on a predicted (e.g., reconstructed) residual {circumflexover (r)}_(t), which in some aspects can be performed in a manner thatis the same as or similar to that described with respect to the residual{circumflex over (r)}_(t) described above with respect to FIG. 7 . Inone illustrative example, residual prediction system 806 (which can bethe same as or similar to the residual prediction system 706 illustratedin FIG. 7 ), can receive an additional input comprising the latentrepresentation z_(p) generated by the object motion prediction system802. For example, decoder network 809 included in residual predictionsystem 806 can concatenate the latent representation z_(p) with thelatent residual representation z_(r), and can generate as output thereconstructed residual {circumflex over (r)}_(t) based on the twoconcatenated latent representations.

An addition operation 810 can be used to add or otherwise combine thepredicted motion compensated frame {tilde over (X)}_(t) and thereconstructed residual {circumflex over (r)}_(t), generating as outputthe reconstructed current frame {circumflex over (X)}_(t).

In one illustrative example, the systems and techniques can compensatecamera motion using rendered optical flow information (e.g., a renderedoptical flow map) generated based on the current camera pose p_(t), thereference camera pose p_(t-1), and the depth map D_(t). For example,rather than providing the camera pose information p_(t) and p_(t-1) asinputs to the warping engine 824 and providing the depth map D_(t) asinput to the encoder network 825 of camera motion prediction system 822,the systems and techniques can generate a rendered optical flow mapusing these same three inputs (e.g., p_(t), p_(t-1), and D_(t)) andprovide the rendered optical flow map as input to the encoder network825 of camera motion prediction system 822.

FIG. 9 is a diagram illustrating an example of a machine learning-basedvideo coding system 900 that utilizes rendered optical flow information(e.g., a rendered optical flow map) generated based on (p_(t), p_(t-1),and D_(t)) as input for generating a camera motion compensated frame X_(t) using a camera motion prediction system 922 and a warping engine924. In one illustrative example, the machine learning-based videocoding system 900 depicted in FIG. 9 can be the same as or similar tothe machine learning-based video coding system 800 depicted in FIG. 8 .For example, the camera motion prediction system 922 and warping engine924 can be the same as or similar to the camera motion prediction system822 and warping engine 824 illustrated in FIG. 8 , the object motionprediction system 902 and warping engine 904 can be the same as orsimilar to the object motion prediction system 802 and warping engine804 illustrated in FIG. 8 , etc.

In the example of FIG. 9 , the rendered optical flow can be generatedbased on p_(t), p_(t-1), and D_(t) by the neural video encoderimplementing the example architecture 900 of FIG. 9 and/or can begenerated by a rendering engine, server, cloud gaming server, or othercomputing device associated with the neural video encoder. In someexamples, the neural video encoder and the computing device used togenerate the rendered optical flow can be included in the same device.In some aspects, the rendered optical flow map can include camera motioninformation that is determined based on the current camera pose p_(t)and the previous camera pose p_(t-1), and the current depth map D_(t).The rendered optical flow can be provided as input to encoder network925 of camera motion prediction system 922, which generates as output alatent representation z_(c) of the rendered optical flow. In someaspects, the latent representation z_(c) can include or otherwise beassociated with camera motion information between frames t and t−1,based at least in part on this camera motion information being includedin the rendered optical flow map.

In some aspects, a neural video decoder implementing some or all ofexample architecture 900 can generate the camera motion compensatedframe X _(t) based on receiving only the latent z_(c) from acorresponding neural video encoder (e.g., a neural video encoder alsoimplementing some or all of example architecture 900). In some cases,the latent z_(c) can be converted to a bitstream and transmitted to thedecoder network 923 and/or camera motion prediction system 922 includedin the neural video decoder. The decoder network 923 can receive thelatent z_(c) as input and generate as output a predicted (e.g.,reconstructed) optical flow map {circumflex over (f)}_(t) ^(cam) for thecurrent frame t. In some aspects, the reconstructed optical flow map{circumflex over (f)}_(t) ^(cam) can be an optical flow map associatedwith motion information determined between the previous frame t−1 andthe current frame t (e.g., the same or similar as the motion informationincluded in the rendered optical flow map determined at the neural videoencoder side). Based on warping engine 924 receiving the reconstructedcamera motion optical flow map {circumflex over (f)}_(t) ^(cam) asinput, warping engine 924 can generate the camera motion compensatedframe X _(t) without receiving the camera pose information p_(t) andp_(t-1) from the neural video encoder (or associated renderingengine/server), as was the case in the example of FIG. 8 .

In some aspects, based on utilizing the rendered optical flow as inputto camera motion prediction system 922, fewer bits may be transmittedbetween the neural video encoder side and the neural video decoder sidefor each respective frame of video data (e.g., rendered video dataand/or gaming content video data), as transmitting only the latentrepresentation z_(c) of the rendered optical flow map may utilize fewerbits than transmitting the latent representation of the depth map D_(t)and one or more of the camera poses p_(t) and p_(t-1) (e.g., as was thecase with respect to FIG. 8 ).

In some examples, the machine learning-based video coding systems 800and 900, illustrated and described above with respect to FIGS. 8 and 9respectively, can generate a reconstructed frame {circumflex over(X)}_(t) based on performing an accumulative warping for the cameramotion compensated frame X _(t) and the object motion compensated frame{tilde over (X)}_(t) (e.g., accumulative based on the object motioncompensated frame {tilde over (X)}_(t) being generated based onreceiving the previously warped the camera motion compensated frame X_(t) as input). In some aspects, the systems and techniques can generatethe reconstructed frame {circumflex over (X)}_(t) based on performingcamera motion compensation and object motion compensation separately,without performing an accumulative warping between the camera motioncompensated result and the object motion compensated result.

For example, FIG. 10 is a diagram illustrating an example of a machinelearning-based video coding system 1000 that can be used to generate thereconstructed frame {circumflex over (X)}_(t) based on performingoptical flow compression, iterative flow estimation, and residual flowestimation, without performing accumulative warping. In one illustrativeexample, the machine learning-based video coding system 1000 depicted inFIG. 10 can be the same as or similar to one or more of the machinelearning-based video coding system 900 illustrated in FIG. 9 and/or themachine learning-based video coding system 800 illustrated in FIG. 8 .For example, the camera motion prediction system 1022 and warping engine1024 can be the same as or similar to one or more of the camera motionprediction systems 922 and/or 824 and the warping engines 924 and/or 824illustrated in FIGS. 9 and 8 respectively; the object motion predictionsystem 1002 and warping engine 1004 can be the same as or similar to oneor more of the object motion prediction systems 902 and/or 802 and thewarping engines 904 and/or 804 illustrated in FIGS. 9 and 8 ,respectively; etc.

In the example of FIG. 10 , a rendered optical flow can be generatedbased on the camera poses p_(t) and p_(t-1), and the depth map D_(t) aswas described above with respect to FIG. 9 . As illustrated, the cameramotion prediction system 1022 can receive as input the rendered opticalflow map, in a same or similar manner as was described with respect tothe camera motion prediction system 922 illustrated in FIG. 9 . Thecamera motion prediction system 1022 may additionally generate as outputa reconstructed camera motion optical flow map {circumflex over (f)}_(t)^(cam) that is the same as or similar to the reconstructed camera motionoptical flow map generated by camera motion prediction system 922illustrated in FIG. 9 . The warping engine 1024 may receive as input thepreviously reconstructed (e.g., reference) frame {circumflex over(X)}_(t-1) and the reconstructed camera motion optical flow map{circumflex over (f)}_(t) ^(cam), which can be the same as or similar tothat described above with respect to warping engine 924 illustrated inFIG. 9 .

As depicted in FIG. 10 , the output generated by the warping engine 1024(e.g., the camera motion compensated frame X _(t)) can be provided asinput only to the object motion prediction system 1002 (e.g., ratherthan being provided as input to both the object motion prediction system1002 and the warping engine 1004, as was depicted with respect to FIG. 9and the object motion prediction system 902 and warping engine 904). Asillustrated in FIG. 10 , the reconstructed camera motion optical flowmap {circumflex over (f)}_(t) ^(cam) generated by the camera motionprediction system 1022 can additionally be provided as an input to theobject motion prediction system 1002.

In one illustrative example, the object motion prediction system 1002can receive as input the currently coded frame X_(t) and the cameramotion compensated frame X _(t) (e.g., the same as or similar to asdescribed above with respect to the object motion prediction systems 802and 902 illustrated in FIGS. 8 and 9 , respectively). Additionally, theobject motion prediction system 1002 can receive as input thereconstructed camera motion optical flow map {circumflex over (f)}_(t)^(cam) (e.g., from camera motion prediction system 1022) and thepreviously reconstructed reference frame {circumflex over (X)}_(t-1).

In some aspects, the example architecture 1000 illustrated in FIG. 10can be used to correct the camera motion compensation/camera motionoptical flow {circumflex over (f)}_(t) ^(cam) directly, prior toperforming a second warp (e.g., the warp performed by warping engine1004) to add the object motion compensation on top of the already warpedcamera motion compensated frame X _(t). For example, the warping engine1004 does not receive the camera motion compensated frame X _(t)directly, and does not apply a warping based on the object motionoptical flow {circumflex over (f)}_(t) ^(obj) to the camera motioncompensated frame X _(t) (e.g., as in FIGS. 8 and 9 ). Instead, thecamera motion compensated frame X _(t) is provided as input to theobject motion prediction system 1002, which generates an object motionoptical map {circumflex over (f)}_(t) ^(obj) that includes both theobject motion optical flow information and camera motion optical flowcorrection information. For example, the object motion prediction system1002 can generate the corrected camera motion compensated output (e.g.,the portion of the object motion optical map {circumflex over (f)}_(t)^(obj) associated with camera motion optical flow correctioninformation) based on receiving the additional inputs of the previouslyreconstructed reference frame {circumflex over (X)}_(t-1) and the cameramotion compensated frame X _(t).

The output of the object motion prediction system 1002 (e.g., the objectmotion optical map {circumflex over (f)}_(t) ^(obj) including objectmotion optical flow information and camera motion optical flowcorrection information) can be provided as input to the warping engine1004. The warping engine 1004 can generate as output the predictedcurrent frame {tilde over (X)}_(t) by warping the previouslyreconstructed frame {circumflex over (X)}_(t-1) based on the objectmotion optical map {circumflex over (f)}_(t) ^(obj) rather than warpingthe camera motion compensated frame X _(t) based on an object motionoptical map that does not include camera motion optical flowcorrections, as was the case with respect to FIGS. 8 and 9 ). In oneillustrative example, the systems and techniques can utilize the examplearchitecture 1000 illustrated in FIG. 10 to generate an initial cameramotion compensated frame X _(t), correct the initial camera motioncompensated frame X _(t) based on an object motion optical flow map{circumflex over (f)}_(t) ^(obj) that includes object motion opticalflow information and camera motion optical flow correction information,and generate the predicted current frame {tilde over (X)}_(t) by warpingthe previously reconstructed frame {circumflex over (X)}_(t-1) based onthe object motion optical map {circumflex over (f)}_(t) ^(obj) withcamera motion corrections.

FIG. 11 is a flowchart illustrating an example of a process 1100 forprocessing video data. At block 1102, the process 1100 includesobtaining a frame of encoded video data associated with an input frame,the frame of encoded video data including camera information associatedwith generating the video data and a residual. For example, some (orall) of the frame of encoded video data can be obtained from one or moreof the encoder networks 825, 805, and/or 807 illustrated in FIG. 8 . Insome cases, the camera information can include one or more of a cameraprojection matrix, camera pose information associated with the inputframe, and depth information associated with the input frame.

For example, the camera projection matrix can be used to perform 3Dwarping, such as by using the 3D warping engine 824 illustrated in FIG.8 . In some examples, the camera pose information can include camerapose information from one or more times, such as time t and time t−1. Insome examples, the depth information associated with the input frame canbe a depth map, such as the depth map d_(t) illustrated in FIG. 8 . Adepth map can include depth information for a plurality of pixelsincluded in the input frame.

At block 1104, the process 1100 includes generating a camera motioncompensated frame based on a reference frame and the camera information.For example, the camera motion compensated frame can be generated using3D warping engine 824 illustrated in FIG. 8 . In some aspects,generating the camera motion compensated frame comprises warping thereference frame based on a reconstructed depth map associated with theinput frame and the camera information. The camera information used toperform the warping can include camera pose information associated withthe input frame and camera pose information associated with thereference frame.

In some examples, the reconstructed depth map can be generated based ona latent representation of a depth map associated with the input frame,wherein the latent representation of the depth map is generated using anautoencoder. For example, the latent representation of the depth map canbe generated by the depth map prediction system 822 illustrated in FIG.8 , with the encoder network 825 and decoder network 823 included in anautoencoder.

In some examples, the camera motion compensated frame can be generatedbased on warping the reference frame using a reconstructed optical flowmap associated with the camera information. The reconstructed opticalflow map can be generated based on camera motion information that isdetermined from camera pose information and a depth map associated withthe input frame. For example, the camera pose information can includecamera pose information associated with the input frame and can includecamera pose information associated with the reference frame. In someexamples, the reconstructed optical flow map can be generated based on alatent representation of an optical flow map generated using the camerapose information and the depth map.

For example, the latent representation of the optical flow map can bethe same as or similar to the latent z_(o) illustrated in FIG. 8 asbeing generated using the object motion prediction system 802 and/or theencoder network 805. In some cases, the optical flow information can begenerated based on a latent representation of an optical flow associatedwith the input frame. The optical flow information can include objectmotion information determined between the input frame and the referenceframe using an optical flow autoencoder. For example, the optical flowautoencoder can be implemented using the object motion prediction system802 and/or the combination of the encoder network 805 and the decodernetwork 803, each illustrated in FIG. 8 .

At block 1106, the process 1100 includes generating optical flowinformation associated with object motion determined based on at leastthe input frame and the reference frame. For example, the optical flowinformation can be generated using object motion prediction system 802illustrated in FIG. 8 and/or can be generated by one or more of theencoder network 805 and the decoder network 802 also illustrated in FIG.8 . In some cases, the optical flow information can be generated basedon a latent representation of an optical flow associated with the inputframe. The latent representation of the optical flow can be the same asor similar to the latent representation z_(m) illustrated in FIG. 8 . Insome examples, the latent representation of the optical flow associatedwith the input frame can be generated by an encoder network (e.g.,encoder network 805 illustrated in FIG. 8 ) included in an optical flowencoder network and/or optical flow autoencoder. The latentrepresentation of the optical flow can be received by a decoder network(e.g., decoder network 803 illustrated in FIG. 8 ) included in anoptical flow decoder network and/or an optical flow decoder. The opticalflow information can include object motion information determinedbetween the input frame and the reference frame using an optical flowautoencoder.

At block 1108, the process 1100 includes generating a motion compensatedframe by warping the camera motion compensated frame based on theoptical flow information. For example, the motion compensated frame canbe a frame prediction generated by the warping engine 804 illustrated inFIG. 8 , based on receiving as input a camera motion compensated frame(e.g., initial frame prediction) from the 3D warping engine 824 andoptical flow information from object motion prediction system802/decoder network 803, each illustrated in FIG. 8 . The motioncompensated frame can include separate camera motion compensation andobject motion compensation and can be used to generate a reconstructedinput frame.

At block 1110, the process 1100 includes generating, based on a motioncompensated frame and the residual, a reconstructed input frame. Forexample, the motion compensated frame can be generated as describedabove with respect to block 1108 and added with a predicted residualgenerated by a residual prediction system 806 as illustrated in FIG. 8 .The residual can be a predicted residual generated by a decoder network809 of a residual prediction system 806. The predicted residual can begenerated by decoder network 809 based on receiving as input a latentrepresentation z_(r) of the residual, from an encoder network 807, asillustrated in FIG. 8 .

In some examples, the processes described herein (e.g., process 1100and/or other process described herein) may be performed by a computingdevice or apparatus, such as a computing device having the computingdevice architecture 1200 shown in FIG. 12 . The computing device caninclude any suitable device, such as an autonomous vehicle computer, arobotic device, a mobile device (e.g., a mobile phone), a desktopcomputing device, a tablet computing device, a wearable device, a serverdevice, a video game device, an extended reality device (e.g., a virtualreality (VR) device, an augmented reality (AR) device, or a mixedreality (MR) device), a camera device, a set-top box device, and/or anyother computing device with the resource capabilities to perform theprocesses described herein, including process 1100. In some examples,the computing device can include a mobile device, a wearable device, anXR device, a personal computer, a laptop computer, a video server, atelevision, a camera, a set-top box, a video game console, or otherdevice. In some examples, the process 1100 can be performed by acomputing device with the computing device architecture 1200implementing the machine learning-based video coding system of FIG. 8 .

In some cases, the computing device or apparatus may include variouscomponents, such as one or more input devices, one or more outputdevices, one or more processors, one or more microprocessors, one ormore microcomputers, one or more transmitters, receivers or combinedtransmitter-receivers (e.g., referred to as transceivers), one or morecameras, one or more sensors, and/or other component(s) that areconfigured to carry out the steps of processes described herein. In someexamples, the computing device may include a display, a networkinterface configured to communicate and/or receive the data, anycombination thereof, and/or other component(s). The network interfacemay be configured to communicate and/or receive Internet Protocol (IP)based data or other type of data.

The components of the computing device can be implemented in circuitry.For example, the components can include and/or can be implemented usingelectronic circuits or other electronic hardware, which can include oneor more programmable electronic circuits (e.g., microprocessors,graphics processing units (GPUs), digital signal processors (DSPs),central processing units (CPUs), neural processing units (NPUs),application-specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), and/or other suitable electronic circuits), and/orcan include and/or be implemented using computer software, firmware, orany combination thereof, to perform the various operations describedherein.

The process 1100 is illustrated as a logical flow diagram, theoperations of which represents a sequence of operations that can beimplemented in hardware, computer instructions, or a combinationthereof. In the context of computer instructions, the operationsrepresent computer-executable instructions stored on one or morecomputer-readable storage media that, when executed by one or moreprocessors, perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,components, data structures, and the like that perform particularfunctions or implement particular data types. The order in which theoperations are described is not intended to be construed as alimitation, and any number of the described operations can be combinedin any order and/or in parallel to implement the processes.

Additionally, the processes described herein (including process 1100)may be performed under the control of one or more computer systemsconfigured with executable instructions and may be implemented as code(e.g., executable instructions, one or more computer programs, or one ormore applications) executing collectively on one or more processors, byhardware, or combinations thereof. As noted above, the code may bestored on a computer-readable or machine-readable storage medium, forexample, in the form of a computer program comprising a plurality ofinstructions executable by one or more processors. The computer-readableor machine-readable storage medium may be non-transitory.

FIG. 12 illustrates an example computing device architecture 1200 of anexample computing device which can implement the various techniquesdescribed herein. In some examples, the computing device can include amobile device, a wearable device, an XR device, a personal computer, alaptop computer, a video server, a video game console, a robotic device,a set-top box, a television, a camera, a server, or other device. Forexample, the computing device architecture 1200 can implement the neuralP-frame coding system 800 of FIG. 8 . The components of computing devicearchitecture 1200 are shown in electrical communication with each otherusing connection 1205, such as a bus. The example computing devicearchitecture 1200 includes a processing unit (CPU or processor) 1210 andcomputing device connection 1205 that couples various computing devicecomponents including computing device memory 1215, such as read onlymemory (ROM) 1220 and random access memory (RAM) 1225, to processor1210.

Computing device architecture 1200 can include a cache of high-speedmemory connected directly with, in close proximity to, or integrated aspart of processor 1210. Computing device architecture 1200 can copy datafrom memory 1215 and/or the storage device 1230 to cache 1212 for quickaccess by processor 1210. In this way, the cache can provide aperformance boost that avoids processor 1210 delays while waiting fordata. These and other modules can control or be configured to controlprocessor 1210 to perform various actions. Other computing device memory1215 may be available for use as well. Memory 1215 can include multipledifferent types of memory with different performance characteristics.Processor 1210 can include any general purpose processor and a hardwareor software service, such as service 1 1232, service 2 1234, and service3 1236 stored in storage device 1230, configured to control processor1210 as well as a special-purpose processor where software instructionsare incorporated into the processor design. Processor 1210 may be aself-contained system, containing multiple cores or processors, a bus,memory controller, cache, etc. A multi-core processor may be symmetricor asymmetric.

To enable user interaction with the computing device architecture 1200,input device 1245 can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech and so forth.Output device 1235 can also be one or more of a number of outputmechanisms known to those of skill in the art, such as a display,projector, television, speaker device, etc. In some instances,multimodal computing devices can enable a user to provide multiple typesof input to communicate with computing device architecture 1200.Communication interface 1240 can generally govern and manage the userinput and computing device output. There is no restriction on operatingon any particular hardware arrangement and therefore the basic featureshere may easily be substituted for improved hardware or firmwarearrangements as they are developed.

Storage device 1230 is a non-volatile memory and can be a hard disk orother types of computer readable media which can store data that areaccessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs) 1225, read only memory (ROM) 1220, andhybrids thereof. Storage device 1230 can include services 1232, 1234,1236 for controlling processor 1210. Other hardware or software modulesare contemplated. Storage device 1230 can be connected to the computingdevice connection 1205. In one aspect, a hardware module that performs aparticular function can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as processor 1210, connection 1205, output device 1235,and so forth, to carry out the function.

The term “device” is not limited to one or a specific number of physicalobjects (such as one smartphone, one controller, one processing system,and so on). As used herein, a device can include any electronic devicewith one or more parts that may implement at least some portions of thisdisclosure. While the description and examples use the term “device” todescribe various aspects of this disclosure, the term “device” is notlimited to a specific configuration, type, or number of objects.Additionally, the term “system” is not limited to multiple components orspecific examples. For example, a system may be implemented on one ormore printed circuit boards or other substrates, and may have movable orstatic components. While the description and examples use the term“system” to describe various aspects of this disclosure, the term“system” is not limited to a specific configuration, type, or number ofobjects.

Specific details are provided in the description to provide a thoroughunderstanding of the aspects and examples provided herein. However, itwill be understood by one of ordinary skill in the art that the aspectsmay be practiced without these specific details. For clarity ofexplanation, in some instances the present technology may be presentedas including individual functional blocks including functional blockscomprising devices, device components, steps or routines in a methodembodied in software, or combinations of hardware and software.Additional components may be used other than those shown in the figuresand/or described herein. For example, circuits, systems, networks,processes, and other components may be shown as components in blockdiagram form in order not to obscure the aspects in unnecessary detail.In other instances, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the examples.

Individual aspects and/or examples may be described above as a processor method which is depicted as a flowchart, a flow diagram, a data flowdiagram, a structure diagram, or a block diagram. Although a flowchartmay describe the operations as a sequential process, many of theoperations can be performed in parallel or concurrently. In addition,the order of the operations may be re-arranged. A process is terminatedwhen its operations are completed, but could have additional steps notincluded in a figure. A process may correspond to a method, a function,a procedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

Processes and methods according to the above-described examples can beimplemented using computer-executable instructions that are stored orotherwise available from computer-readable media. Such instructions caninclude, for example, instructions and data which cause or otherwiseconfigure a general-purpose computer, special purpose computer, or aprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware,source code, etc.

The term “computer-readable medium” includes, but is not limited to,portable or non-portable storage devices, optical storage devices, andvarious other mediums capable of storing, containing, or carryinginstruction(s) and/or data. A computer-readable medium may include anon-transitory medium in which data can be stored and that does notinclude carrier waves and/or transitory electronic signals propagatingwirelessly or over wired connections. Examples of a non-transitorymedium may include, but are not limited to, a magnetic disk or tape,optical storage media such as flash memory, memory or memory devices,magnetic or optical disks, flash memory, USB devices provided withnon-volatile memory, networked storage devices, compact disk (CD) ordigital versatile disk (DVD), any suitable combination thereof, amongothers. A computer-readable medium may have stored thereon code and/ormachine-executable instructions that may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a class, or any combination of instructions, datastructures, or program statements. A code segment may be coupled toanother code segment or a hardware circuit by passing and/or receivinginformation, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, etc. may be passed, forwarded,or transmitted via any suitable means including memory sharing, messagepassing, token passing, network transmission, or the like.

In some aspects, the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Devices implementing processes and methods according to thesedisclosures can include hardware, software, firmware, middleware,microcode, hardware description languages, or any combination thereof,and can take any of a variety of form factors. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the necessary tasks (e.g., a computer-programproduct) may be stored in a computer-readable or machine-readablemedium. A processor(s) may perform the necessary tasks. Typical examplesof form factors include laptops, smart phones, mobile phones, tabletdevices or other small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are example means for providing the functionsdescribed in the disclosure.

In the foregoing description, aspects of the application are describedwith reference to specific examples thereof, but those skilled in theart will recognize that the application is not limited thereto. Thus,while illustrative examples of the application have been described indetail herein, it is to be understood that the inventive concepts may beotherwise variously embodied and employed, and that the appended claimsare intended to be construed to include such variations, except aslimited by the prior art. Various features and aspects of theabove-described application may be used individually or jointly.Further, aspects of the present disclosure can be utilized in any numberof environments and applications beyond those described herein withoutdeparting from the scope of the specification. The specification anddrawings are, accordingly, to be regarded as illustrative rather thanrestrictive. For the purposes of illustration, methods were described ina particular order. It should be appreciated that in alternate examples,the methods may be performed in a different order than that described.

One of ordinary skill will appreciate that the less than (“<”) andgreater than (“>”) symbols or terminology used herein can be replacedwith less than or equal to (“≤”) and greater than or equal to (“≥”)symbols, respectively, without departing from the scope of thisdescription.

Where components are described as being “configured to” perform certainoperations, such configuration can be accomplished, for example, bydesigning electronic circuits or other hardware to perform theoperation, by programming programmable electronic circuits (e.g.,microprocessors, or other suitable electronic circuits) to perform theoperation, or any combination thereof.

The phrase “coupled to” refers to any component that is physicallyconnected to another component either directly or indirectly, and/or anycomponent that is in communication with another component (e.g.,connected to the other component over a wired or wireless connection,and/or other suitable communication interface) either directly orindirectly.

Claim language or other language reciting “at least one of” a set and/or“one or more” of a set indicates that one member of the set or multiplemembers of the set (in any combination) satisfy the claim. For example,claim language reciting “at least one of A and B” or “at least one of Aor B” means A, B, or A and B. In another example, claim languagereciting “at least one of A, B, and C” or “at least one of A, B, or C”means A, B, C, or A and B, or A and C, or B and C, or A and B and C. Thelanguage “at least one of” a set and/or “one or more” of a set does notlimit the set to the items listed in the set. For example, claimlanguage reciting “at least one of A and B” or “at least one of A or B”can mean A, B, or A and B, and can additionally include items not listedin the set of A and B.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the examples disclosedherein may be implemented as electronic hardware, computer software,firmware, or combinations thereof. To clearly illustrate thisinterchangeability of hardware and software, various illustrativecomponents, blocks, modules, circuits, and steps have been describedabove generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present application.

The techniques described herein may also be implemented in electronichardware, computer software, firmware, or any combination thereof. Suchtechniques may be implemented in any of a variety of devices such asgeneral purposes computers, wireless communication device handsets, orintegrated circuit devices having multiple uses including application inwireless communication device handsets and other devices. Any featuresdescribed as modules or components may be implemented together in anintegrated logic device or separately as discrete but interoperablelogic devices. If implemented in software, the techniques may berealized at least in part by a computer-readable data storage mediumcomprising program code including instructions that, when executed,performs one or more of the methods described above. Thecomputer-readable data storage medium may form part of a computerprogram product, which may include packaging materials. Thecomputer-readable medium may comprise memory or data storage media, suchas random-access memory (RAM) such as synchronous dynamic random accessmemory (SDRAM), read-only memory (ROM), non-volatile random accessmemory (NVRAM), electrically erasable programmable read-only memory(EEPROM), FLASH memory, magnetic or optical data storage media, and thelike. The techniques additionally, or alternatively, may be realized atleast in part by a computer-readable communication medium that carriesor communicates program code in the form of instructions or datastructures and that can be accessed, read, and/or executed by acomputer, such as propagated signals or waves.

The program code may be executed by a processor, which may include oneor more processors, such as one or more digital signal processors(DSPs), general purpose microprocessors, an application specificintegrated circuits (ASICs), field programmable logic arrays (FPGAs), orother equivalent integrated or discrete logic circuitry. Such aprocessor may be configured to perform any of the techniques describedin this disclosure. A general purpose processor may be a microprocessor;but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Accordingly, the term “processor,” as used herein mayrefer to any of the foregoing structure, any combination of theforegoing structure, or any other structure or apparatus suitable forimplementation of the techniques described herein.

Illustrative aspects of the disclosure include:

Aspect 1: An apparatus for decoding video data, the apparatuscomprising: at least one memory; and at least one processor coupled tothe at least one memory, the at least one processor configured to:obtain a frame of encoded video data associated with an input frame, theframe of encoded video data including camera information associated withgenerating the video data and a residual; generate a camera motioncompensated frame based on a reference frame and the camera information;generate optical flow information associated with object motiondetermined based on at least the input frame and the reference frame;generate a motion compensated frame by warping the camera motioncompensated frame based on the optical flow information; and generate,based on the motion compensated frame and the residual, a reconstructedinput frame.

Aspect 2: The apparatus of Aspect 1, wherein the camera informationincludes one or more of a camera projection matrix, a camera poseinformation associated with the input frame, and a depth informationassociated with the input frame.

Aspect 3: The apparatus of Aspect 2, wherein the depth informationassociated with the input frame is a depth map including depthinformation for a plurality of pixels included in the input frame.

Aspect 4: The apparatus of any of Aspects 1 to 3, wherein to generatethe camera motion compensated frame, the at least one processor isconfigured to: warp the reference frame based on a reconstructed depthmap associated with the input frame and the camera information; whereinthe camera information includes camera pose information associated withthe input frame, and camera pose information associated with thereference frame.

Aspect 5: The apparatus of Aspect 4, wherein the at least one processoris further configured to generate the reconstructed depth map based on alatent representation of a depth map associated with the input frame,wherein the latent representation of the depth map is generated using anautoencoder.

Aspect 6: The apparatus of any of Aspects 4 to 5, wherein: the opticalflow information is generated based on a latent representation of anoptical flow associated with the input frame; and the optical flowinformation includes object motion information determined between theinput frame and the reference frame using an optical flow autoencoder.

Aspect 7: The apparatus of any of Aspects 1 to 6, wherein to generatethe camera motion compensated frame, the at least one processor isconfigured to: warp the reference frame based on a reconstructed opticalflow map associated with the camera information; wherein thereconstructed optical flow map is generated based on camera motioninformation determined using camera pose information and a depth mapassociated with the input frame.

Aspect 8: The apparatus of Aspect 7, wherein the camera pose informationincludes camera pose information associated with the input frame andcamera pose information associated with the reference frame.

Aspect 9: The apparatus of Aspect 8, wherein the at least one processoris further configured to generate the reconstructed optical flow mapbased on a latent representation of an optical map generated using thecamera pose information and the depth map.

Aspect 10: The apparatus of any of Aspects 7 to 9, wherein: the opticalflow information is generated based on a latent representation of anoptical flow associated with the input frame; and the optical flowinformation includes object motion information determined between theinput frame and the reference frame using an optical flow autoencoder.

Aspect 11: A method for decoding video data, the method comprising:obtaining a frame of encoded video data associated with an input frame,the frame of encoded video data including camera information associatedwith generating the video data and a residual; generating a cameramotion compensated frame based on a reference frame and the camerainformation; generating optical flow information associated with objectmotion determined based on at least the input frame and the referenceframe; generating a motion compensated frame by warping the cameramotion compensated frame based on the optical flow information; andgenerating, based on the motion compensated frame and the residual, areconstructed input frame.

Aspect 12: The method of Aspect 11, wherein the camera informationincludes one or more of a camera projection matrix, a camera poseinformation associated with the input frame, and a depth informationassociated with the input frame.

Aspect 13: The method of Aspect 12, wherein the depth informationassociated with the input frame is a depth map including depthinformation for a plurality of pixels included in the input frame.

Aspect 14: The method of any of Aspects 11 to 13, wherein generating thecamera motion compensated frame comprises warping the reference framebased on a reconstructed depth map associated with the input frame andthe camera information, wherein the camera information includes camerapose information associated with the input frame, and camera poseinformation associated with the reference frame.

Aspect 15: The method of Aspect 14, further comprising generating thereconstructed depth map based on a latent representation of a depth mapassociated with the input frame, wherein the latent representation ofthe depth map is generated using an autoencoder.

Aspect 16: The method of any of Aspects 14 to 15, wherein: the opticalflow information is generated based on a latent representation of anoptical flow associated with the input frame; and the optical flowinformation includes object motion information determined between theinput frame and the reference frame using an optical flow autoencoder.

Aspect 17: The method of any of Aspects 11 to 16, wherein generating thecamera motion compensated frame comprises warping the reference framebased on a reconstructed optical flow map associated with the camerainformation, wherein the reconstructed optical flow map is generatedbased on camera motion information determined using camera poseinformation and a depth map associated with the input frame.

Aspect 18: The method of Aspect 17, wherein the camera pose informationincludes camera pose information associated with the input frame andcamera pose information associated with the reference frame.

Aspect 19: The method of Aspect 18, further comprising generating thereconstructed optical flow map based on a latent representation of anoptical map generated using the camera pose information and the depthmap.

Aspect 20: The method of any of Aspects 17 to 19, wherein: the opticalflow information is generated based on a latent representation of anoptical flow associated with the input frame; and the optical flowinformation includes object motion information determined between theinput frame and the reference frame using an optical flow autoencoder.

Aspect 21: An apparatus for encoding video data, the apparatuscomprising: at least one memory; and at least one processor coupled tothe at least one memory, the at least one processor configured to:obtain an input frame of video data and camera information associatedwith generating the input frame of video data; generate a camera motioncompensated frame based on a reference frame and the camera information;generate optical flow information associated with object motiondetermined based on at least the input frame and the reference frame;generate a motion compensated frame by warping the camera motioncompensated frame based on the optical flow information; determine,based on a difference between the input frame and a reconstructed inputframe generated using the motion compensated frame, a residual; andgenerate a frame of encoded video data associated with the input frameof video data, the frame of encoded video data including the camerainformation and the residual.

Aspect 22: The apparatus of Aspect 21, wherein the camera informationincludes one or more of a camera projection matrix, a camera poseinformation associated with the input frame, and a depth informationassociated with the input frame.

Aspect 23: The apparatus of Aspect 22, wherein the depth informationassociated with the input frame is a depth map including depthinformation for a plurality of pixels included in the input frame.

Aspect 24: The apparatus of any of Aspects 21 to 23, wherein to generatethe camera motion compensated frame, the at least one processor isconfigured to: warp the reference frame based on a reconstructed depthmap associated with the input frame and the camera information; whereinthe camera information includes camera pose information associated withthe input frame, and camera pose information associated with thereference frame.

Aspect 25: The apparatus of Aspect 24, wherein the at least oneprocessor is further configured to generate the reconstructed depth mapbased on a latent representation of a depth map associated with theinput frame, wherein the latent representation of the depth map isgenerated using an autoencoder.

Aspect 26: The apparatus of any of Aspects 24 to 25, wherein: theoptical flow information is generated based on a latent representationof an optical flow associated with the input frame; and the optical flowinformation includes object motion information determined between theinput frame and the reference frame using an optical flow autoencoder.

Aspect 27: The apparatus of any of Aspects 21 to 26, wherein to generatethe camera motion compensated frame, the at least one processor isconfigured to: warp the reference frame based on a reconstructed opticalflow map associated with the camera information; wherein thereconstructed optical flow map is generated based on camera motioninformation determined using camera pose information and a depth mapassociated with the input frame.

Aspect 28: The apparatus of Aspect 27, wherein the at least oneprocessor is further configured to generate the reconstructed opticalflow map based on a latent representation of an optical map generatedusing the camera pose information and the depth map.

Aspect 29: The apparatus of any of Aspects 27 to 28, wherein: theoptical flow information is generated based on a latent representationof an optical flow associated with the input frame; and the optical flowinformation includes object motion information determined between theinput frame and the reference frame using an optical flow autoencoder.

Aspect 30: A method for encoding video data, the method comprising:obtaining an input frame of video data and camera information associatedwith generating the input frame of video data; generating a cameramotion compensated frame based on a reference frame and the camerainformation; generating optical flow information associated with objectmotion determined based on at least the input frame and the referenceframe; generating a motion compensated frame by warping the cameramotion compensated frame based on the optical flow information;determining, based on a difference between the input frame and areconstructed input frame generated using the motion compensated frame,a residual; and generating a frame of encoded video data associated withthe input frame of video data, the frame of encoded video data includingthe camera information and the residual.

Aspect 31: A non-transitory computer-readable storage medium havingstored thereon instructions which, when executed by one or moreprocessors, cause the one or more processors to perform any of theoperations of Aspects 1 to 20.

Aspect 32: A non-transitory computer-readable storage medium havingstored thereon instructions which, when executed by one or moreprocessors, cause the one or more processors to perform any of theoperations of Aspects 21 to 30.

Aspect 33: An apparatus comprising means for performing any of theoperations of Aspects 1 to 20.

Aspect 34: An apparatus comprising means for performing any of theoperations of Aspects 21 to 30.

What is claimed is:
 1. An apparatus for decoding video data, theapparatus comprising: at least one memory; and at least one processorcoupled to the at least one memory, the at least one processorconfigured to: obtain a frame of encoded video data associated with aninput frame, the frame of encoded video data including camerainformation associated with generating the video data and a residual;generate a camera motion compensated frame based on a reference frameand the camera information; generate optical flow information associatedwith object motion determined based on at least the input frame and thereference frame; generate a motion compensated frame by warping thecamera motion compensated frame based on the optical flow information;and generate, based on the motion compensated frame and the residual, areconstructed input frame.
 2. The apparatus of claim 1, wherein thecamera information includes one or more of a camera projection matrix, acamera pose information associated with the input frame, and a depthinformation associated with the input frame.
 3. The apparatus of claim2, wherein the depth information associated with the input frame is adepth map including depth information for a plurality of pixels includedin the input frame.
 4. The apparatus of claim 1, wherein to generate thecamera motion compensated frame, the at least one processor isconfigured to: warp the reference frame based on a reconstructed depthmap associated with the input frame and the camera information; whereinthe camera information includes camera pose information associated withthe input frame, and camera pose information associated with thereference frame.
 5. The apparatus of claim 4, wherein the at least oneprocessor is further configured to generate the reconstructed depth mapbased on a latent representation of a depth map associated with theinput frame, wherein the latent representation of the depth map isgenerated using an autoencoder.
 6. The apparatus of claim 4, wherein:the optical flow information is generated based on a latentrepresentation of an optical flow associated with the input frame; andthe optical flow information includes object motion informationdetermined between the input frame and the reference frame using anoptical flow autoencoder.
 7. The apparatus of claim 1, wherein togenerate the camera motion compensated frame, the at least one processoris configured to: warp the reference frame based on a reconstructedoptical flow map associated with the camera information; wherein thereconstructed optical flow map is generated based on camera motioninformation determined using camera pose information and a depth mapassociated with the input frame.
 8. The apparatus of claim 7, whereinthe camera pose information includes camera pose information associatedwith the input frame and camera pose information associated with thereference frame.
 9. The apparatus of claim 8, wherein the at least oneprocessor is further configured to generate the reconstructed opticalflow map based on a latent representation of an optical flow mapgenerated using the camera pose information and the depth map.
 10. Theapparatus of claim 7, wherein: the optical flow information is generatedbased on a latent representation of an optical flow associated with theinput frame; and the optical flow information includes object motioninformation determined between the input frame and the reference frameusing an optical flow autoencoder.
 11. A method for decoding video data,the method comprising: obtaining a frame of encoded video dataassociated with an input frame, the frame of encoded video dataincluding camera information associated with generating the video dataand a residual; generating a camera motion compensated frame based on areference frame and the camera information; generating optical flowinformation associated with object motion determined based on at leastthe input frame and the reference frame; generating a motion compensatedframe by warping the camera motion compensated frame based on theoptical flow information; and generating, based on the motioncompensated frame and the residual, a reconstructed input frame.
 12. Themethod of claim 11, wherein the camera information includes one or moreof a camera projection matrix, a camera pose information associated withthe input frame, and a depth information associated with the inputframe.
 13. The method of claim 12, wherein the depth informationassociated with the input frame is a depth map including depthinformation for a plurality of pixels included in the input frame. 14.The method of claim 11, wherein generating the camera motion compensatedframe comprises warping the reference frame based on a reconstructeddepth map associated with the input frame and the camera information,wherein the camera information includes camera pose informationassociated with the input frame, and camera pose information associatedwith the reference frame.
 15. The method of claim 14, further comprisinggenerating the reconstructed depth map based on a latent representationof a depth map associated with the input frame, wherein the latentrepresentation of the depth map is generated using an autoencoder. 16.The method of claim 14, wherein: the optical flow information isgenerated based on a latent representation of an optical flow associatedwith the input frame; and the optical flow information includes objectmotion information determined between the input frame and the referenceframe using an optical flow autoencoder.
 17. The method of claim 11,wherein generating the camera motion compensated frame comprises warpingthe reference frame based on a reconstructed optical flow map associatedwith the camera information, wherein the reconstructed optical flow mapis generated based on camera motion information determined using camerapose information and a depth map associated with the input frame. 18.The method of claim 17, wherein the camera pose information includescamera pose information associated with the input frame and camera poseinformation associated with the reference frame.
 19. The method of claim18, further comprising generating the reconstructed optical flow mapbased on a latent representation of an optical flow map generated usingthe camera pose information and the depth map.
 20. The method of claim17, wherein: the optical flow information is generated based on a latentrepresentation of an optical flow associated with the input frame; andthe optical flow information includes object motion informationdetermined between the input frame and the reference frame using anoptical flow autoencoder.
 21. An apparatus for encoding video data, theapparatus comprising: at least one memory; and at least one processorcoupled to the at least one memory, the at least one processorconfigured to: obtain an input frame of video data and camerainformation associated with generating the input frame of video data;generate a camera motion compensated frame based on a reference frameand the camera information; generate optical flow information associatedwith object motion determined based on at least the input frame and thereference frame; generate a motion compensated frame by warping thecamera motion compensated frame based on the optical flow information;determine, based on a difference between the input frame and areconstructed input frame generated using the motion compensated frame,a residual; and generate a frame of encoded video data associated withthe input frame of video data, the frame of encoded video data includingthe camera information and the residual.
 22. The apparatus of claim 21,wherein the camera information includes one or more of a cameraprojection matrix, a camera pose information associated with the inputframe, and a depth information associated with the input frame.
 23. Theapparatus of claim 22, wherein the depth information associated with theinput frame is a depth map including depth information for a pluralityof pixels included in the input frame.
 24. The apparatus of claim 21,wherein to generate the camera motion compensated frame, the at leastone processor is configured to: warp the reference frame based on areconstructed depth map associated with the input frame and the camerainformation; wherein the camera information includes camera poseinformation associated with the input frame, and camera pose informationassociated with the reference frame.
 25. The apparatus of claim 24,wherein the at least one processor is further configured to generate thereconstructed depth map based on a latent representation of a depth mapassociated with the input frame, wherein the latent representation ofthe depth map is generated using an autoencoder.
 26. The apparatus ofclaim 24, wherein: the optical flow information is generated based on alatent representation of an optical flow associated with the inputframe; and the optical flow information includes object motioninformation determined between the input frame and the reference frameusing an optical flow autoencoder.
 27. The apparatus of claim 21,wherein to generate the camera motion compensated frame, the at leastone processor is configured to: warp the reference frame based on areconstructed optical flow map associated with the camera information;wherein the reconstructed optical flow map is generated based on cameramotion information determined using camera pose information and a depthmap associated with the input frame.
 28. The apparatus of claim 27,wherein the at least one processor is further configured to generate thereconstructed optical flow map based on a latent representation of anoptical flow map generated using the camera pose information and thedepth map.
 29. The apparatus of claim 27, wherein: the optical flowinformation is generated based on a latent representation of an opticalflow associated with the input frame; and the optical flow informationincludes object motion information determined between the input frameand the reference frame using an optical flow autoencoder.
 30. A methodfor encoding video data, the method comprising: obtaining an input frameof video data and camera information associated with generating theinput frame of video data; generating a camera motion compensated framebased on a reference frame and the camera information; generatingoptical flow information associated with object motion determined basedon at least the input frame and the reference frame; generating a motioncompensated frame by warping the camera motion compensated frame basedon the optical flow information; determining, based on a differencebetween the input frame and a reconstructed input frame generated usingthe motion compensated frame, a residual; and generating a frame ofencoded video data associated with the input frame of video data, theframe of encoded video data including the camera information and theresidual.